2024-03-17T23:25:04.789 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-17T23:25:04.789 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} --archive /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463 --name pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463/orig.config.yaml 2024-03-17T23:25:04.842 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-17T23:25:04.929 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463 branch: wip-batrick-testing-20240314.183243 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7608463' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - mons down, quorum - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.dir_split install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: c382027d5c66ef4e0aa1131ea3e8080c658d3184 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: c382027d5c66ef4e0aa1131ea3e8080c658d3184 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGa961Dzqp+tZg2LmYxpfomuSbZshpA1HtJSYIX3CuvgME0iQN+ao9/HIY45qeXnip8dzYNNh0gXIiveDWyCb7Q= smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF13kXw2LFa5gBTd0GORJflZaTdCJwAX+2CxqrDbryHFFpnX5TRr98ypsxSC4IQ6EKun6Sou+S9WFod5Q33Okd4= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKly/HsdlgRfddttfzA8lT1U9Iem8vnkeqm4JoxnTqWbKLFAN+rgu0xjXOGOauZyeTP5ihoV4/mKET/uSXzx7rc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-17_23:17:27 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-17T23:25:04.929 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa; will attempt to use it 2024-03-17T23:25:04.930 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks 2024-03-17T23:25:04.930 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-17T23:25:04.931 INFO:teuthology.task.internal:Checking packages... 2024-03-17T23:25:04.955 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-17T23:25:04.955 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-17T23:25:04.955 INFO:teuthology.packaging:ref: None 2024-03-17T23:25:04.955 INFO:teuthology.packaging:tag: None 2024-03-17T23:25:04.955 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-17T23:25:04.955 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:25:04.955 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-17T23:25:05.096 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-17T23:25:05.097 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-17T23:25:05.104 INFO:teuthology.task.internal:no buildpackages task found 2024-03-17T23:25:05.104 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-17T23:25:05.110 INFO:teuthology.task.internal:Saving configuration 2024-03-17T23:25:05.135 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-17T23:25:05.141 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-17T23:25:05.166 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-17 23:20:11.734961', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGa961Dzqp+tZg2LmYxpfomuSbZshpA1HtJSYIX3CuvgME0iQN+ao9/HIY45qeXnip8dzYNNh0gXIiveDWyCb7Q='} 2024-03-17T23:25:05.187 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-17 23:20:11.731987', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF13kXw2LFa5gBTd0GORJflZaTdCJwAX+2CxqrDbryHFFpnX5TRr98ypsxSC4IQ6EKun6Sou+S9WFod5Q33Okd4='} 2024-03-17T23:25:05.210 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-17 23:20:11.733405', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKly/HsdlgRfddttfzA8lT1U9Iem8vnkeqm4JoxnTqWbKLFAN+rgu0xjXOGOauZyeTP5ihoV4/mKET/uSXzx7rc='} 2024-03-17T23:25:05.210 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-17T23:25:05.216 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-17T23:25:05.217 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-17T23:25:05.217 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-17T23:25:05.217 INFO:teuthology.run_tasks:Running task console_log... 2024-03-17T23:25:05.311 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff8d9dfa160>, signals=[15]) 2024-03-17T23:25:05.311 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-17T23:25:05.317 INFO:teuthology.task.internal:Opening connections... 2024-03-17T23:25:05.317 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2024-03-17T23:25:05.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:25:05.430 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-03-17T23:25:05.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:25:05.519 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-03-17T23:25:05.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:25:05.611 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-17T23:25:05.617 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2024-03-17T23:25:05.636 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2024-03-17T23:25:05.636 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2024-03-17T23:25:05.694 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Stream" 2024-03-17T23:25:05.697 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="9" 2024-03-17T23:25:05.697 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2024-03-17T23:25:05.697 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2024-03-17T23:25:05.697 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="9" 2024-03-17T23:25:05.697 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el9" 2024-03-17T23:25:05.697 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-17T23:25:05.697 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2024-03-17T23:25:05.697 INFO:teuthology.orchestra.run.smithi012.stdout:LOGO="fedora-logo-icon" 2024-03-17T23:25:05.697 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-17T23:25:05.697 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/" 2024-03-17T23:25:05.697 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-17T23:25:05.697 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-17T23:25:05.698 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-17T23:25:05.698 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2024-03-17T23:25:05.723 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-03-17T23:25:05.740 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-03-17T23:25:05.741 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-03-17T23:25:05.798 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2024-03-17T23:25:05.798 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="9" 2024-03-17T23:25:05.798 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2024-03-17T23:25:05.798 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2024-03-17T23:25:05.798 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="9" 2024-03-17T23:25:05.798 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el9" 2024-03-17T23:25:05.798 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-17T23:25:05.798 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2024-03-17T23:25:05.799 INFO:teuthology.orchestra.run.smithi079.stdout:LOGO="fedora-logo-icon" 2024-03-17T23:25:05.799 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-17T23:25:05.799 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2024-03-17T23:25:05.799 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-17T23:25:05.799 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-17T23:25:05.799 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-17T23:25:05.799 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-03-17T23:25:05.831 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-03-17T23:25:05.849 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-03-17T23:25:05.849 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-03-17T23:25:05.906 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-03-17T23:25:05.906 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-03-17T23:25:05.906 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-03-17T23:25:05.906 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-03-17T23:25:05.906 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-03-17T23:25:05.906 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-03-17T23:25:05.906 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-17T23:25:05.907 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-03-17T23:25:05.907 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-03-17T23:25:05.907 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-17T23:25:05.907 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-03-17T23:25:05.907 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-17T23:25:05.907 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-17T23:25:05.907 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-17T23:25:05.907 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-03-17T23:25:05.931 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-17T23:25:05.942 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-17T23:25:05.948 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-17T23:25:05.948 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2024-03-17T23:25:05.950 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-03-17T23:25:05.952 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-03-17T23:25:05.968 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-17T23:25:05.975 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-17T23:25:05.975 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2024-03-17T23:25:06.008 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-03-17T23:25:06.011 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-03-17T23:25:06.171 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-17T23:25:06.252 INFO:teuthology.run_tasks:Running task kernel... 2024-03-17T23:25:06.266 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-17T23:25:06.266 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-17T23:25:06.266 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-17T23:25:06.267 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-17T23:25:06.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:25:06.284 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-17T23:25:06.497 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3845b7ac715a3ff198582638d199a90e30e254da 2024-03-17T23:25:06.498 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3845b7ac715a3ff198582638d199a90e30e254da"... 2024-03-17T23:25:06.498 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-03-17T23:25:06.515 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-361.el9.x86_64 2024-03-17T23:25:06.516 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3845b7ac715a3ff198582638d199a90e30e254da 2024-03-17T23:25:06.516 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-17T23:25:06.516 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum upgrade -y linux-firmware 2024-03-17T23:25:08.588 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-17T23:25:10.072 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-17T23:25:11.162 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-17T23:25:28.060 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-17T23:25:32.802 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 6.1 MB 00:01 2024-03-17T23:25:37.452 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 4.5 kB/s | 16 kB 00:03 2024-03-17T23:25:43.128 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 3.7 MB/s | 21 MB 00:05 2024-03-17T23:25:48.211 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-17T23:25:50.615 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-17T23:25:50.625 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-17T23:25:50.625 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-17T23:25:50.625 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-17T23:25:50.625 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2024-03-17T23:25:50.625 INFO:teuthology.orchestra.run.smithi012.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-17T23:25:50.625 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:25:50.625 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-17T23:25:50.625 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-17T23:25:50.625 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 1 Package 2024-03-17T23:25:50.625 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:25:50.625 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 375 M 2024-03-17T23:25:50.625 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-17T23:25:57.644 INFO:teuthology.orchestra.run.smithi012.stdout:linux-firmware-20240219-143.el9.noarch.rpm 53 MB/s | 375 MB 00:07 2024-03-17T23:25:57.645 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-17T23:25:57.645 INFO:teuthology.orchestra.run.smithi012.stdout:Total 53 MB/s | 375 MB 00:07 2024-03-17T23:25:57.646 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-17T23:25:57.742 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-17T23:25:57.743 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-17T23:26:00.258 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-17T23:26:00.259 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-17T23:26:06.707 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-17T23:26:06.948 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-17T23:26:07.046 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-17T23:26:08.367 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-17T23:26:08.367 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-17T23:26:08.770 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-17T23:26:08.770 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:26:08.771 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2024-03-17T23:26:08.771 INFO:teuthology.orchestra.run.smithi012.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-17T23:26:08.771 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:26:08.771 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-17T23:26:09.071 INFO:teuthology.task.kernel:Downloading kernel 3845b7ac715a3ff198582638d199a90e30e254da on client.0... 2024-03-17T23:26:09.072 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3845b7ac715a3ff198582638d199a90e30e254da 2024-03-17T23:26:09.218 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/ 2024-03-17T23:26:09.219 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc7_g3845b7ac715a-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-17T23:26:09.221 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi012.front.sepia.ceph.com to complete... 2024-03-17T23:26:27.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17 23:26:27 URL:https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/kernel-6.8.0_rc7_g3845b7ac715a-1.x86_64.rpm [336075563/336075563] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-17T23:26:27.473 INFO:teuthology.orchestra.run.smithi012.stderr:FINISHED --2024-03-17 23:26:27-- 2024-03-17T23:26:27.473 INFO:teuthology.orchestra.run.smithi012.stderr:Total wall clock time: 18s 2024-03-17T23:26:27.473 INFO:teuthology.orchestra.run.smithi012.stderr:Downloaded: 1 files, 321M in 18s (17.9 MB/s) 2024-03-17T23:26:27.476 INFO:teuthology.task.kernel:Installing kernel 3845b7ac715a3ff198582638d199a90e30e254da on client.0... 2024-03-17T23:26:27.476 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-17T23:26:29.509 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying... ######################################## 2024-03-17T23:26:29.564 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing... ######################################## 2024-03-17T23:26:29.648 INFO:teuthology.orchestra.run.smithi012.stdout:Updating / installing... 2024-03-17T23:26:37.596 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-6.8.0_rc7_g3845b7ac715a-1 ######################################## 2024-03-17T23:26:53.815 INFO:teuthology.orchestra.run.smithi012.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-17T23:27:16.993 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-17T23:27:16.993 DEBUG:teuthology.orchestra.run.smithi012:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-17T23:27:17.083 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a 2024-03-17T23:27:17.083 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/System.map 2024-03-17T23:27:17.083 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/config 2024-03-17T23:27:17.083 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel 2024-03-17T23:27:17.084 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch 2024-03-17T23:27:17.084 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86 2024-03-17T23:27:17.084 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto 2024-03-17T23:27:17.084 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-17T23:27:17.084 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-17T23:27:17.084 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-17T23:27:17.084 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-17T23:27:17.084 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-17T23:27:17.084 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-17T23:27:17.084 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-17T23:27:17.084 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-17T23:27:17.084 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-17T23:27:17.084 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-17T23:27:17.085 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-17T23:27:17.085 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-17T23:27:17.085 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-17T23:27:17.085 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-17T23:27:17.085 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-17T23:27:17.085 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-17T23:27:17.085 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-17T23:27:17.085 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-17T23:27:17.085 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-17T23:27:17.085 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-17T23:27:17.085 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel 2024-03-17T23:27:17.085 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu 2024-03-17T23:27:17.085 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu/mce 2024-03-17T23:27:17.086 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-17T23:27:17.086 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm 2024-03-17T23:27:17.086 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-17T23:27:17.086 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-17T23:27:17.086 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm.ko 2024-03-17T23:27:17.086 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto 2024-03-17T23:27:17.086 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/ansi_cprng.ko 2024-03-17T23:27:17.086 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/anubis.ko 2024-03-17T23:27:17.086 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/arc4.ko 2024-03-17T23:27:17.086 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx 2024-03-17T23:27:17.086 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_memcpy.ko 2024-03-17T23:27:17.086 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_pq.ko 2024-03-17T23:27:17.086 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-17T23:27:17.087 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_tx.ko 2024-03-17T23:27:17.087 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_xor.ko 2024-03-17T23:27:17.087 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/raid6test.ko 2024-03-17T23:27:17.087 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/authenc.ko 2024-03-17T23:27:17.087 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/authencesn.ko 2024-03-17T23:27:17.087 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blake2b_generic.ko 2024-03-17T23:27:17.087 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blowfish_common.ko 2024-03-17T23:27:17.087 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blowfish_generic.ko 2024-03-17T23:27:17.087 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/camellia_generic.ko 2024-03-17T23:27:17.087 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast5_generic.ko 2024-03-17T23:27:17.087 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast6_generic.ko 2024-03-17T23:27:17.087 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast_common.ko 2024-03-17T23:27:17.087 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/ccm.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cmac.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crc32_generic.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crypto_engine.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crypto_user.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cts.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/des_generic.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/dh_generic.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/echainiv.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/essiv.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/fcrypt.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/khazad.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/lz4.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/lz4hc.ko 2024-03-17T23:27:17.088 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/md4.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/michael_mic.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/pcbc.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/pcrypt.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/rmd160.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/seed.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/serpent_generic.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/tcrypt.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/tea.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/twofish_common.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/twofish_generic.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/vmac.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/wp512.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xcbc.ko 2024-03-17T23:27:17.089 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xor.ko 2024-03-17T23:27:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xxhash_generic.ko 2024-03-17T23:27:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers 2024-03-17T23:27:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi 2024-03-17T23:27:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-17T23:27:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/acpi_pad.ko 2024-03-17T23:27:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/custom_method.ko 2024-03-17T23:27:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/ec_sys.ko 2024-03-17T23:27:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/platform_profile.ko 2024-03-17T23:27:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/sbs.ko 2024-03-17T23:27:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/sbshc.ko 2024-03-17T23:27:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/video.ko 2024-03-17T23:27:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/acard-ahci.ko 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/ahci_platform.ko 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/ata_generic.ko 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/libahci_platform.ko 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_acpi.ko 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ali.ko 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_amd.ko 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_artop.ko 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_atiixp.ko 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_atp867x.ko 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cmd640.ko 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cmd64x.ko 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cypress.ko 2024-03-17T23:27:17.091 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_efar.ko 2024-03-17T23:27:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt366.ko 2024-03-17T23:27:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt37x.ko 2024-03-17T23:27:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-17T23:27:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-17T23:27:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_it8213.ko 2024-03-17T23:27:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_it821x.ko 2024-03-17T23:27:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_jmicron.ko 2024-03-17T23:27:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_marvell.ko 2024-03-17T23:27:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_mpiix.ko 2024-03-17T23:27:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_netcell.ko 2024-03-17T23:27:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ninja32.ko 2024-03-17T23:27:17.092 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ns87410.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ns87415.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_oldpiix.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_opti.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_optidma.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pcmcia.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_piccolo.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_rdc.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sch.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_serverworks.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sil680.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sis.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sl82c105.ko 2024-03-17T23:27:17.093 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_triflex.ko 2024-03-17T23:27:17.094 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_via.ko 2024-03-17T23:27:17.094 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pdc_adma.ko 2024-03-17T23:27:17.094 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_inic162x.ko 2024-03-17T23:27:17.094 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_mv.ko 2024-03-17T23:27:17.094 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_nv.ko 2024-03-17T23:27:17.094 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_promise.ko 2024-03-17T23:27:17.094 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_qstor.ko 2024-03-17T23:27:17.094 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sil.ko 2024-03-17T23:27:17.094 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sil24.ko 2024-03-17T23:27:17.094 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sis.ko 2024-03-17T23:27:17.094 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_svw.ko 2024-03-17T23:27:17.094 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sx4.ko 2024-03-17T23:27:17.094 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_uli.ko 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_via.ko 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_vsc.ko 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/auxdisplay 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/auxdisplay/ks0108.ko 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/bcma 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/bcma/bcma.ko 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/aoe 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/aoe/aoe.ko 2024-03-17T23:27:17.095 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/brd.ko 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/drbd 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/drbd/drbd.ko 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/floppy.ko 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/loop.ko 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/mtip32xx 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/nbd.ko 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/null_blk 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/null_blk/null_blk.ko 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/pktcdvd.ko 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/rbd.ko 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/virtio_blk.ko 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkback 2024-03-17T23:27:17.096 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkfront.ko 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/zram 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/zram/zram.ko 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hangcheck-timer.ko 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/via-rng.ko 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-17T23:27:17.097 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-17T23:27:17.098 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-17T23:27:17.098 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-17T23:27:17.098 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-17T23:27:17.098 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-17T23:27:17.098 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/lp.ko 2024-03-17T23:27:17.098 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/mwave 2024-03-17T23:27:17.098 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/mwave/mwave.ko 2024-03-17T23:27:17.098 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ppdev.ko 2024-03-17T23:27:17.098 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tlclk.ko 2024-03-17T23:27:17.098 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm 2024-03-17T23:27:17.098 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm.ko 2024-03-17T23:27:17.098 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-17T23:27:17.098 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/uv_mmtimer.ko 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/virtio_console.ko 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto 2024-03-17T23:27:17.099 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp/ccp.ko 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_common 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/padlock-aes.ko 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/padlock-sha.ko 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/virtio 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dca 2024-03-17T23:27:17.100 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dca/dca.ko 2024-03-17T23:27:17.101 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/devfreq 2024-03-17T23:27:17.101 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-17T23:27:17.101 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma 2024-03-17T23:27:17.101 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/dw 2024-03-17T23:27:17.101 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-17T23:27:17.101 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/ioat 2024-03-17T23:27:17.101 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-17T23:27:17.101 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac 2024-03-17T23:27:17.101 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/amd64_edac.ko 2024-03-17T23:27:17.101 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/e752x_edac.ko 2024-03-17T23:27:17.101 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/edac_mce_amd.ko 2024-03-17T23:27:17.101 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i3000_edac.ko 2024-03-17T23:27:17.101 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i3200_edac.ko 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i5100_edac.ko 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i5400_edac.ko 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i7300_edac.ko 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i7core_edac.ko 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i82975x_edac.ko 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/ie31200_edac.ko 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/sb_edac.ko 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/x38_edac.ko 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware/edd.ko 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio/gpio-ich.ko 2024-03-17T23:27:17.102 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-appleir.ko 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-aureal.ko 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-axff.ko 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-dr.ko 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-elecom.ko 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-elo.ko 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-emsff.ko 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gaff.ko 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gt683r.ko 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gyration.ko 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-17T23:27:17.103 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtekff.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-hyperv.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-icade.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-keytouch.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-kye.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-lcpower.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-led.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-lenovo.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-multitouch.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-ortek.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-penmount.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-petalynx.ko 2024-03-17T23:27:17.104 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-picolcd.ko 2024-03-17T23:27:17.105 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-pl.ko 2024-03-17T23:27:17.105 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-plantronics.ko 2024-03-17T23:27:17.105 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-primax.ko 2024-03-17T23:27:17.105 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-rmi.ko 2024-03-17T23:27:17.105 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-17T23:27:17.105 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-common.ko 2024-03-17T23:27:17.105 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-17T23:27:17.105 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-17T23:27:17.105 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-17T23:27:17.105 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-17T23:27:17.105 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-17T23:27:17.105 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-17T23:27:17.106 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-17T23:27:17.106 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-17T23:27:17.106 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-17T23:27:17.106 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat.ko 2024-03-17T23:27:17.106 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-saitek.ko 2024-03-17T23:27:17.106 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-samsung.ko 2024-03-17T23:27:17.106 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-17T23:27:17.106 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sjoy.ko 2024-03-17T23:27:17.106 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sony.ko 2024-03-17T23:27:17.106 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-speedlink.ko 2024-03-17T23:27:17.106 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-steelseries.ko 2024-03-17T23:27:17.106 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sunplus.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-tivo.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-tmff.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-topseed.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-twinhan.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-uclogic.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-waltop.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-wiimote.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-xinmo.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-zpff.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-zydacron.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/uhid.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/wacom.ko 2024-03-17T23:27:17.107 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_balloon.ko 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_utils.ko 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_vmbus.ko 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/abituguru.ko 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/abituguru3.ko 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ad7414.ko 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ad7418.ko 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adc128d818.ko 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1025.ko 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1026.ko 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1029.ko 2024-03-17T23:27:17.108 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1031.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm9240.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ads7828.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7410.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7411.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7462.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7470.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7475.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7x10.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/amc6821.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/applesmc.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asb100.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asc7621.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-17T23:27:17.109 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/atxp1.ko 2024-03-17T23:27:17.110 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/coretemp.ko 2024-03-17T23:27:17.110 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-17T23:27:17.110 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/dme1737.ko 2024-03-17T23:27:17.110 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ds1621.ko 2024-03-17T23:27:17.110 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ds620.ko 2024-03-17T23:27:17.110 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/emc1403.ko 2024-03-17T23:27:17.110 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/emc6w201.ko 2024-03-17T23:27:17.110 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f71805f.ko 2024-03-17T23:27:17.110 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f71882fg.ko 2024-03-17T23:27:17.110 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f75375s.ko 2024-03-17T23:27:17.110 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/fam15h_power.ko 2024-03-17T23:27:17.110 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/fschmd.ko 2024-03-17T23:27:17.110 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/g760a.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/g762.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/gl518sm.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/gl520sm.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/i5500_temp.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/i5k_amb.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ibmaem.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ibmpex.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ina209.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ina2xx.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/it87.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/k10temp.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/k8temp.ko 2024-03-17T23:27:17.111 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lineage-pem.ko 2024-03-17T23:27:17.112 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm63.ko 2024-03-17T23:27:17.112 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm73.ko 2024-03-17T23:27:17.112 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm75.ko 2024-03-17T23:27:17.112 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm77.ko 2024-03-17T23:27:17.112 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm78.ko 2024-03-17T23:27:17.112 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm80.ko 2024-03-17T23:27:17.112 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm83.ko 2024-03-17T23:27:17.112 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm85.ko 2024-03-17T23:27:17.112 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm87.ko 2024-03-17T23:27:17.112 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm90.ko 2024-03-17T23:27:17.112 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm92.ko 2024-03-17T23:27:17.112 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm93.ko 2024-03-17T23:27:17.113 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95234.ko 2024-03-17T23:27:17.113 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95241.ko 2024-03-17T23:27:17.113 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95245.ko 2024-03-17T23:27:17.113 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc2945.ko 2024-03-17T23:27:17.113 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4151.ko 2024-03-17T23:27:17.113 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4215.ko 2024-03-17T23:27:17.113 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4222.ko 2024-03-17T23:27:17.113 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4245.ko 2024-03-17T23:27:17.113 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4260.ko 2024-03-17T23:27:17.113 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4261.ko 2024-03-17T23:27:17.113 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max16065.ko 2024-03-17T23:27:17.113 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max1619.ko 2024-03-17T23:27:17.113 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max1668.ko 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max197.ko 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6639.ko 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6650.ko 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6697.ko 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/mcp3021.ko 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6683.ko 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6775-core.ko 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6775.ko 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct7802.ko 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pc87360.ko 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pc87427.ko 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pcf8591.ko 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus 2024-03-17T23:27:17.114 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/powr1220.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch5627.ko 2024-03-17T23:27:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch5636.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sht15.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sht21.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/shtc1.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sis5595.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47b397.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47m1.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47m192.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/thmc50.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp102.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp103.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp401.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp421.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/via-cputemp.ko 2024-03-17T23:27:17.116 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/via686a.ko 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/vt1211.ko 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/vt8231.ko 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83627ehf.ko 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83627hf.ko 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83781d.ko 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83791d.ko 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83792d.ko 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83793.ko 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83795.ko 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83l785ts.ko 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83l786ng.ko 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos 2024-03-17T23:27:17.117 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-17T23:27:17.118 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-17T23:27:17.118 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses 2024-03-17T23:27:17.118 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-17T23:27:17.118 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-17T23:27:17.118 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-17T23:27:17.118 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-17T23:27:17.118 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-17T23:27:17.118 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-17T23:27:17.118 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-17T23:27:17.118 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-17T23:27:17.118 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-17T23:27:17.118 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-dev.ko 2024-03-17T23:27:17.119 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-mux.ko 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-smbus.ko 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-stub.ko 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_core.ko 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw 2024-03-17T23:27:17.120 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/cxgb4 2024-03-17T23:27:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-17T23:27:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx4 2024-03-17T23:27:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-17T23:27:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx5 2024-03-17T23:27:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-17T23:27:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mthca 2024-03-17T23:27:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-17T23:27:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp 2024-03-17T23:27:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/ipoib 2024-03-17T23:27:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-17T23:27:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/iser 2024-03-17T23:27:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-17T23:27:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/isert 2024-03-17T23:27:17.122 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-17T23:27:17.122 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srp 2024-03-17T23:27:17.122 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-17T23:27:17.122 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srpt 2024-03-17T23:27:17.122 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-17T23:27:17.122 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input 2024-03-17T23:27:17.122 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport 2024-03-17T23:27:17.122 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-17T23:27:17.122 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-17T23:27:17.122 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/gameport.ko 2024-03-17T23:27:17.122 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/joydev.ko 2024-03-17T23:27:17.122 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/keyboard 2024-03-17T23:27:17.122 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/appletouch.ko 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/bcm5974.ko 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/cyapatp.ko 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/sermouse.ko 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/rmi4 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/altera_ps2.ko 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/arc_ps2.ko 2024-03-17T23:27:17.123 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/serio_raw.ko 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/sparse-keymap.ko 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-blinkm.ko 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-lm3530.ko 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-lp3944.ko 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-ss4200.ko 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-17T23:27:17.124 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/bcache 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/bcache/bcache.ko 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-bio-prison.ko 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-cache-smq.ko 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-cache.ko 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-crypt.ko 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-delay.ko 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-flakey.ko 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-log-userspace.ko 2024-03-17T23:27:17.125 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-multipath.ko 2024-03-17T23:27:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-queue-length.ko 2024-03-17T23:27:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-raid.ko 2024-03-17T23:27:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-round-robin.ko 2024-03-17T23:27:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-service-time.ko 2024-03-17T23:27:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-switch.ko 2024-03-17T23:27:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-thin-pool.ko 2024-03-17T23:27:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-verity.ko 2024-03-17T23:27:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/persistent-data 2024-03-17T23:27:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-17T23:27:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid0.ko 2024-03-17T23:27:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid1.ko 2024-03-17T23:27:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid10.ko 2024-03-17T23:27:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid456.ko 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptbase.ko 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptctl.ko 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptfc.ko 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptlan.ko 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptsas.ko 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptscsih.ko 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptspi.ko 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/lpc_ich.ko 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/lpc_sch.ko 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/mfd-core.ko 2024-03-17T23:27:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/sm501.ko 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/viperboard.ko 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/vx855.ko 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/wl1273-core.ko 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/altera-stapl 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/apds9802als.ko 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/apds990x.ko 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/bh1770glc.ko 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/cb710 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/cb710/cb710.ko 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/at24.ko 2024-03-17T23:27:17.128 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/max6875.ko 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/enclosure.ko 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/hpilo.ko 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/isl29003.ko 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/isl29020.ko 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei-me.ko 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei-txe.ko 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei.ko 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-gru 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-17T23:27:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp 2024-03-17T23:27:17.130 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-17T23:27:17.130 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-17T23:27:17.130 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-17T23:27:17.130 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tifm_7xx1.ko 2024-03-17T23:27:17.130 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tifm_core.ko 2024-03-17T23:27:17.130 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tsl2550.ko 2024-03-17T23:27:17.130 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_balloon.ko 2024-03-17T23:27:17.130 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_vmci 2024-03-17T23:27:17.130 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-17T23:27:17.130 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net 2024-03-17T23:27:17.130 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/bonding 2024-03-17T23:27:17.130 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/bonding/bonding.ko 2024-03-17T23:27:17.130 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/b53 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6xxx 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dummy.ko 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/eql.ko 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-17T23:27:17.131 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/adaptec 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/agere 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/alteon 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/altera 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-17T23:27:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/alx 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-17T23:27:17.133 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom 2024-03-17T23:27:17.134 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-17T23:27:17.134 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-17T23:27:17.134 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-17T23:27:17.134 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-17T23:27:17.134 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-17T23:27:17.134 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/genet 2024-03-17T23:27:17.134 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-17T23:27:17.134 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-17T23:27:17.134 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade 2024-03-17T23:27:17.134 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade/bna 2024-03-17T23:27:17.134 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-17T23:27:17.134 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio 2024-03-17T23:27:17.134 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-17T23:27:17.135 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-17T23:27:17.135 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-17T23:27:17.135 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-17T23:27:17.135 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-17T23:27:17.135 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-17T23:27:17.135 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-17T23:27:17.135 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-17T23:27:17.135 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-17T23:27:17.135 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-17T23:27:17.135 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco 2024-03-17T23:27:17.135 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco/enic 2024-03-17T23:27:17.135 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dnet.ko 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex 2024-03-17T23:27:17.136 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex/benet 2024-03-17T23:27:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-17T23:27:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ethoc.ko 2024-03-17T23:27:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/fealnx.ko 2024-03-17T23:27:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel 2024-03-17T23:27:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-17T23:27:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000 2024-03-17T23:27:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-17T23:27:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000e 2024-03-17T23:27:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-17T23:27:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/fm10k 2024-03-17T23:27:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-17T23:27:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/i40e 2024-03-17T23:27:17.137 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-17T23:27:17.138 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/iavf 2024-03-17T23:27:17.138 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-17T23:27:17.138 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igb 2024-03-17T23:27:17.138 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-17T23:27:17.138 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igbvf 2024-03-17T23:27:17.138 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-17T23:27:17.138 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-17T23:27:17.138 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-17T23:27:17.138 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-17T23:27:17.138 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-17T23:27:17.138 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/jme.ko 2024-03-17T23:27:17.138 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/micrel 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom 2024-03-17T23:27:17.139 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/neterion 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/nvidia 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-17T23:27:17.140 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rdc 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rocker 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sfc 2024-03-17T23:27:17.141 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-17T23:27:17.142 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/silan 2024-03-17T23:27:17.142 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-17T23:27:17.142 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis 2024-03-17T23:27:17.142 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-17T23:27:17.142 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-17T23:27:17.142 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc 2024-03-17T23:27:17.142 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-17T23:27:17.142 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-17T23:27:17.142 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-17T23:27:17.142 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-17T23:27:17.142 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro 2024-03-17T23:27:17.142 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/tehuti 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ti 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-17T23:27:17.143 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-17T23:27:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet 2024-03-17T23:27:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-17T23:27:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-17T23:27:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/xircom 2024-03-17T23:27:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-17T23:27:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/geneve.ko 2024-03-17T23:27:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/hyperv 2024-03-17T23:27:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-17T23:27:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ieee802154 2024-03-17T23:27:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-17T23:27:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ifb.ko 2024-03-17T23:27:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ipvlan 2024-03-17T23:27:17.144 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/macvlan.ko 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/macvtap.ko 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio.ko 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mii.ko 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/net_failover.ko 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/netconsole.ko 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/nlmon.ko 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-17T23:27:17.145 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-17T23:27:17.146 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy 2024-03-17T23:27:17.146 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/amd.ko 2024-03-17T23:27:17.146 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-17T23:27:17.146 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-17T23:27:17.146 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-17T23:27:17.146 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm87xx.ko 2024-03-17T23:27:17.146 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/broadcom.ko 2024-03-17T23:27:17.146 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/cicada.ko 2024-03-17T23:27:17.146 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/davicom.ko 2024-03-17T23:27:17.146 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/dp83640.ko 2024-03-17T23:27:17.146 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/et1011c.ko 2024-03-17T23:27:17.146 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/icplus.ko 2024-03-17T23:27:17.146 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/lxt.ko 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/marvell.ko 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/micrel.ko 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/national.ko 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/phylink.ko 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/qsemi.ko 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/realtek.ko 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/smsc.ko 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/ste10Xp.ko 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/vitesse.ko 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_async.ko 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-17T23:27:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pppoe.ko 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pppox.ko 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pptp.ko 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip/slhc.ko 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip/slip.ko 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/sungem_phy.ko 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/tap.ko 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team.ko 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-17T23:27:17.148 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_random.ko 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/tun.ko 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/veth.ko 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/virtio_net.ko 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vmxnet3 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vxlan 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vxlan/vxlan.ko 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netback 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netfront.ko 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ntb 2024-03-17T23:27:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ntb/ntb.ko 2024-03-17T23:27:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme 2024-03-17T23:27:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host 2024-03-17T23:27:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host/nvme-core.ko 2024-03-17T23:27:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host/nvme.ko 2024-03-17T23:27:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport 2024-03-17T23:27:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport.ko 2024-03-17T23:27:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_cs.ko 2024-03-17T23:27:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_pc.ko 2024-03-17T23:27:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_serial.ko 2024-03-17T23:27:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci 2024-03-17T23:27:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/hotplug 2024-03-17T23:27:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-17T23:27:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/pcie 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/xen-pcifront.ko 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/i82092.ko 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/pd6729.ko 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl/intel 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/acerhdf.ko 2024-03-17T23:27:17.151 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/hdaps.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel_ips.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-17T23:27:17.152 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/wmi.ko 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-17T23:27:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps_parport.ko 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/pps_core.ko 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp/ptp.ko 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp/ptp_kvm.ko 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-17T23:27:17.154 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-em3027.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-max6900.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-17T23:27:17.155 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-x1205.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-9xxx.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-sas.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-xxxx.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/BusLogic.ko 2024-03-17T23:27:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/a100u2w.ko 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aacraid 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/advansys.ko 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/am53c974.ko 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/arcmsr 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/atp870u.ko 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/be2iscsi 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bfa 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bfa/bfa.ko 2024-03-17T23:27:17.157 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2fc 2024-03-17T23:27:17.158 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-17T23:27:17.158 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2i 2024-03-17T23:27:17.158 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-17T23:27:17.158 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ch.ko 2024-03-17T23:27:17.158 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/csiostor 2024-03-17T23:27:17.158 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-17T23:27:17.158 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi 2024-03-17T23:27:17.158 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-17T23:27:17.158 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-17T23:27:17.158 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-17T23:27:17.158 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-17T23:27:17.158 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-17T23:27:17.158 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/dc395x.ko 2024-03-17T23:27:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler 2024-03-17T23:27:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-17T23:27:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-17T23:27:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-17T23:27:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-17T23:27:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/dmx3191d.ko 2024-03-17T23:27:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esas2r 2024-03-17T23:27:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-17T23:27:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esp_scsi.ko 2024-03-17T23:27:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe 2024-03-17T23:27:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-17T23:27:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-17T23:27:17.159 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fnic 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fnic/fnic.ko 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hpsa.ko 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hptiop.ko 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hv_storvsc.ko 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/initio.ko 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ipr.ko 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ips.ko 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/isci 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/isci/isci.ko 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libfc 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libfc/libfc.ko 2024-03-17T23:27:17.160 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libiscsi.ko 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libsas 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libsas/libsas.ko 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/lpfc 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid.ko 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mpt3sas 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvsas 2024-03-17T23:27:17.161 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvumi.ko 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pm8001 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pmcraid.ko 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla1280.ko 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla4xxx 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/raid_class.ko 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_debug.ko 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-17T23:27:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ses.ko 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/st.ko 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/stex.ko 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/sym53c8xx_2 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/virtio_scsi.ko 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/wd719x.ko 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ssb 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ssb/ssb.ko 2024-03-17T23:27:17.163 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/iscsi 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/loopback 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_file.ko 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_iblock.ko 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_mod.ko 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_pscsi.ko 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_user.ko 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/tcm_fc 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal 2024-03-17T23:27:17.164 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-17T23:27:17.165 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/ipwireless 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/n_gsm.ko 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/n_hdlc.ko 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/nozomi.ko 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/8250 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/arc_uart.ko 2024-03-17T23:27:17.166 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/jsm 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/core 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/host 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio.ko 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_aec.ko 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_cif.ko 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_pci_generic.ko 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_sercos3.ko 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb 2024-03-17T23:27:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/cdc-acm.ko 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/usblp.ko 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/usbtmc.ko 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/host 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image/mdc800.ko 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image/microtek.ko 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/adutux.ko 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/appledisplay.ko 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/emi26.ko 2024-03-17T23:27:17.168 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/emi62.ko 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/ezusb.ko 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/idmouse.ko 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/iowarrior.ko 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/ldusb.ko 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/legousbtower.ko 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/sisusbvga 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usb3503.ko 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usblcd.ko 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/uss720.ko 2024-03-17T23:27:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/yurex.ko 2024-03-17T23:27:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/phy 2024-03-17T23:27:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/phy/phy-generic.ko 2024-03-17T23:27:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial 2024-03-17T23:27:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/aircable.ko 2024-03-17T23:27:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ark3116.ko 2024-03-17T23:27:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-17T23:27:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ch341.ko 2024-03-17T23:27:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cp210x.ko 2024-03-17T23:27:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cyberjack.ko 2024-03-17T23:27:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-17T23:27:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-17T23:27:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/empeg.ko 2024-03-17T23:27:17.170 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/io_ti.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ipaq.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ipw.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ir-usb.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/keyspan.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mct_u232.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mos7720.ko 2024-03-17T23:27:17.171 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mos7840.ko 2024-03-17T23:27:17.172 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/navman.ko 2024-03-17T23:27:17.172 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/omninet.ko 2024-03-17T23:27:17.172 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/opticon.ko 2024-03-17T23:27:17.172 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/option.ko 2024-03-17T23:27:17.172 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/oti6858.ko 2024-03-17T23:27:17.172 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/pl2303.ko 2024-03-17T23:27:17.172 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/qcaux.ko 2024-03-17T23:27:17.172 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/qcserial.ko 2024-03-17T23:27:17.172 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/quatech2.ko 2024-03-17T23:27:17.172 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/safe_serial.ko 2024-03-17T23:27:17.172 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/sierra.ko 2024-03-17T23:27:17.172 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-17T23:27:17.172 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ssu100.ko 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/symbolserial.ko 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb_debug.ko 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/visor.ko 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/whiteheat.ko 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/uas.ko 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-17T23:27:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-17T23:27:17.174 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-17T23:27:17.174 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-17T23:27:17.174 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-17T23:27:17.174 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-karma.ko 2024-03-17T23:27:17.174 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-17T23:27:17.174 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-17T23:27:17.174 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-17T23:27:17.174 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-17T23:27:17.174 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-17T23:27:17.174 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/usb-storage.ko 2024-03-17T23:27:17.174 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip 2024-03-17T23:27:17.174 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-17T23:27:17.174 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/vfio.ko 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost.ko 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_net.ko 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_scsi.ko 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video 2024-03-17T23:27:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/apple_bl.ko 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/lcd.ko 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio.ko 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_balloon.ko 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_mmio.ko 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci.ko 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_ring.ko 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters 2024-03-17T23:27:17.176 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters/ds2482.ko 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters/ds2490.ko 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-17T23:27:17.177 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/wire.ko 2024-03-17T23:27:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog 2024-03-17T23:27:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-17T23:27:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-17T23:27:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-17T23:27:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/hpwdt.ko 2024-03-17T23:27:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/i6300esb.ko 2024-03-17T23:27:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-17T23:27:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-17T23:27:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ib700wdt.ko 2024-03-17T23:27:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ibmasr.ko 2024-03-17T23:27:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-17T23:27:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-17T23:27:17.178 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/it87_wdt.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/machzwd.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/nv_tco.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/softdog.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/via_wdt.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/wdt_pci.ko 2024-03-17T23:27:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/xen_wdt.ko 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-evtchn.ko 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-gntalloc.ko 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-gntdev.ko 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-pciback 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-privcmd.ko 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xenfs 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/9p 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/9p/9p.ko 2024-03-17T23:27:17.180 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/affs 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/affs/affs.ko 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/befs 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/befs/befs.ko 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/binfmt_misc.ko 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/btrfs 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/btrfs/btrfs.ko 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cachefiles 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cachefiles/cachefiles.ko 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ceph 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ceph/ceph.ko 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/coda 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/coda/coda.ko 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cramfs 2024-03-17T23:27:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cramfs/cramfs.ko 2024-03-17T23:27:17.182 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/dlm 2024-03-17T23:27:17.182 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/dlm/dlm.ko 2024-03-17T23:27:17.182 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ecryptfs 2024-03-17T23:27:17.182 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-17T23:27:17.182 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/f2fs 2024-03-17T23:27:17.182 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/f2fs/f2fs.ko 2024-03-17T23:27:17.182 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat 2024-03-17T23:27:17.182 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/fat.ko 2024-03-17T23:27:17.182 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/msdos.ko 2024-03-17T23:27:17.182 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/vfat.ko 2024-03-17T23:27:17.182 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse 2024-03-17T23:27:17.182 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse/cuse.ko 2024-03-17T23:27:17.182 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse/fuse.ko 2024-03-17T23:27:17.183 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/gfs2 2024-03-17T23:27:17.183 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/gfs2/gfs2.ko 2024-03-17T23:27:17.183 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfs 2024-03-17T23:27:17.183 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfs/hfs.ko 2024-03-17T23:27:17.183 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfsplus 2024-03-17T23:27:17.183 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfsplus/hfsplus.ko 2024-03-17T23:27:17.183 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/isofs 2024-03-17T23:27:17.183 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/isofs/isofs.ko 2024-03-17T23:27:17.183 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/jfs 2024-03-17T23:27:17.183 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/jfs/jfs.ko 2024-03-17T23:27:17.183 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/lockd 2024-03-17T23:27:17.183 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/lockd/lockd.ko 2024-03-17T23:27:17.183 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/minix 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/minix/minix.ko 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/netfs 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/netfs/netfs.ko 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/blocklayout 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/filelayout 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/flexfilelayout 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfs.ko 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfsv3.ko 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfsv4.ko 2024-03-17T23:27:17.184 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common/grace.ko 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common/nfs_acl.ko 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfsd 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfsd/nfsd.ko 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nilfs2 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nilfs2/nilfs2.ko 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-celtic.ko 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-centeuro.ko 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-croatian.ko 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-cyrillic.ko 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-gaelic.ko 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-greek.ko 2024-03-17T23:27:17.185 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-iceland.ko 2024-03-17T23:27:17.186 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-inuit.ko 2024-03-17T23:27:17.186 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-roman.ko 2024-03-17T23:27:17.186 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-romanian.ko 2024-03-17T23:27:17.186 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-turkish.ko 2024-03-17T23:27:17.186 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1250.ko 2024-03-17T23:27:17.186 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1251.ko 2024-03-17T23:27:17.186 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1255.ko 2024-03-17T23:27:17.186 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp737.ko 2024-03-17T23:27:17.186 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp775.ko 2024-03-17T23:27:17.186 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp850.ko 2024-03-17T23:27:17.186 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp852.ko 2024-03-17T23:27:17.186 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp855.ko 2024-03-17T23:27:17.186 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp857.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp860.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp861.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp862.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp863.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp864.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp865.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp866.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp869.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp874.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp932.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp936.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp949.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp950.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_euc-jp.ko 2024-03-17T23:27:17.187 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-1.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-13.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-14.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-15.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-2.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-3.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-4.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-5.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-6.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-7.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-9.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-r.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-ru.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-u.ko 2024-03-17T23:27:17.188 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-17T23:27:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_utf8.ko 2024-03-17T23:27:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2 2024-03-17T23:27:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/cluster 2024-03-17T23:27:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-17T23:27:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlm 2024-03-17T23:27:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-17T23:27:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlmfs 2024-03-17T23:27:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-17T23:27:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2.ko 2024-03-17T23:27:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-17T23:27:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-17T23:27:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/overlayfs 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/overlayfs/overlay.ko 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/pstore 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/pstore/ramoops.ko 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/reiserfs 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/reiserfs/reiserfs.ko 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/romfs 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/romfs/romfs.ko 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/client 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/client/cifs.ko 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common/cifs_arc4.ko 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common/cifs_md4.ko 2024-03-17T23:27:17.190 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/squashfs 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/squashfs/squashfs.ko 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/sysv 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/sysv/sysv.ko 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/udf 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/udf/udf.ko 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ufs 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ufs/ufs.ko 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/xfs 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/xfs/xfs.ko 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/rcu 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/rcu/rcutorture.ko 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/torture.ko 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/trace 2024-03-17T23:27:17.191 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/asn1_encoder.ko 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crc-itu-t.ko 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crc8.ko 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto/libarc4.ko 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto/libdes.ko 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/libcrc32c.ko 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lru_cache.ko 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4/lz4_compress.ko 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4/lz4hc_compress.ko 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/math 2024-03-17T23:27:17.192 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/math/cordic.ko 2024-03-17T23:27:17.193 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/raid6 2024-03-17T23:27:17.193 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/raid6/raid6_pq.ko 2024-03-17T23:27:17.193 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/reed_solomon 2024-03-17T23:27:17.193 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-17T23:27:17.193 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_bm.ko 2024-03-17T23:27:17.193 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_fsm.ko 2024-03-17T23:27:17.193 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_kmp.ko 2024-03-17T23:27:17.193 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/zstd 2024-03-17T23:27:17.193 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/zstd/zstd_compress.ko 2024-03-17T23:27:17.193 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/mm 2024-03-17T23:27:17.193 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/mm/hwpoison-inject.ko 2024-03-17T23:27:17.193 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net 2024-03-17T23:27:17.193 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan 2024-03-17T23:27:17.194 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/6lowpan.ko 2024-03-17T23:27:17.194 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_dest.ko 2024-03-17T23:27:17.194 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_fragment.ko 2024-03-17T23:27:17.194 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_hop.ko 2024-03-17T23:27:17.194 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-17T23:27:17.194 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_mobility.ko 2024-03-17T23:27:17.194 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_routing.ko 2024-03-17T23:27:17.194 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_udp.ko 2024-03-17T23:27:17.194 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802 2024-03-17T23:27:17.194 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/garp.ko 2024-03-17T23:27:17.194 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/mrp.ko 2024-03-17T23:27:17.194 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/p8022.ko 2024-03-17T23:27:17.194 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/psnap.ko 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/stp.ko 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/8021q 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/8021q/8021q.ko 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet.ko 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_fd.ko 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_rdma.ko 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_virtio.ko 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/batman-adv 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/batman-adv/batman-adv.ko 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/br_netfilter.ko 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/bridge.ko 2024-03-17T23:27:17.195 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-17T23:27:17.196 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtables.ko 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ceph 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ceph/libceph.ko 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core/failover.ko 2024-03-17T23:27:17.197 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core/pktgen.ko 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp.ko 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_diag.ko 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_ipv4.ko 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_ipv6.ko 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dns_resolver 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dns_resolver/dns_resolver.ko 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/dsa_core.ko 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_brcm.ko 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_dsa.ko 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_none.ko 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_trailer.ko 2024-03-17T23:27:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/6lowpan 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/ieee802154.ko 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ah4.ko 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/esp4.ko 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/fou.ko 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/gre.ko 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/inet_diag.ko 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_gre.ko 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_tunnel.ko 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_vti.ko 2024-03-17T23:27:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ipcomp.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ipip.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-17T23:27:17.200 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-17T23:27:17.201 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-17T23:27:17.201 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-17T23:27:17.201 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-17T23:27:17.201 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-17T23:27:17.201 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-17T23:27:17.201 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-17T23:27:17.201 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-17T23:27:17.201 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-17T23:27:17.201 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_bic.ko 2024-03-17T23:27:17.201 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_dctcp.ko 2024-03-17T23:27:17.201 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_diag.ko 2024-03-17T23:27:17.201 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_highspeed.ko 2024-03-17T23:27:17.201 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_htcp.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_hybla.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_illinois.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_lp.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_scalable.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_vegas.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_veno.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_westwood.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_yeah.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tunnel4.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/udp_diag.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/udp_tunnel.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ah6.ko 2024-03-17T23:27:17.202 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/esp6.ko 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/fou6.ko 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_tunnel.ko 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_vti.ko 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ipcomp6.ko 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-17T23:27:17.203 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-17T23:27:17.204 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/sit.ko 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/tunnel6.ko 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/key 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/key/af_key.ko 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_core.ko 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_eth.ko 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ip.ko 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ip6.ko 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_netlink.ko 2024-03-17T23:27:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ppp.ko 2024-03-17T23:27:17.206 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/llc 2024-03-17T23:27:17.206 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/llc/llc.ko 2024-03-17T23:27:17.206 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mac802154 2024-03-17T23:27:17.206 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mac802154/mac802154.ko 2024-03-17T23:27:17.206 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mpls 2024-03-17T23:27:17.206 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mpls/mpls_gso.ko 2024-03-17T23:27:17.206 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter 2024-03-17T23:27:17.206 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset 2024-03-17T23:27:17.206 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set.ko 2024-03-17T23:27:17.206 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-17T23:27:17.206 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-17T23:27:17.206 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-17T23:27:17.206 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-17T23:27:17.207 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conncount.ko 2024-03-17T23:27:17.208 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack.ko 2024-03-17T23:27:17.209 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-17T23:27:17.209 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-17T23:27:17.209 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-17T23:27:17.209 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-17T23:27:17.209 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-17T23:27:17.209 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-17T23:27:17.209 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-17T23:27:17.209 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-17T23:27:17.209 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-17T23:27:17.209 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-17T23:27:17.209 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-17T23:27:17.209 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-17T23:27:17.209 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_flow_table.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_log_syslog.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_irc.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_sip.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_tables.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-17T23:27:17.210 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_log.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_chain_nat.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_compat.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_ct.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_fib.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_hash.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_limit.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_log.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_masq.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_nat.ko 2024-03-17T23:27:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_numgen.ko 2024-03-17T23:27:17.212 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_queue.ko 2024-03-17T23:27:17.212 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_quota.ko 2024-03-17T23:27:17.212 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_redir.ko 2024-03-17T23:27:17.212 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_reject.ko 2024-03-17T23:27:17.212 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_reject_inet.ko 2024-03-17T23:27:17.212 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_socket.ko 2024-03-17T23:27:17.212 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_tunnel.ko 2024-03-17T23:27:17.212 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_AUDIT.ko 2024-03-17T23:27:17.212 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-17T23:27:17.212 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-17T23:27:17.212 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-17T23:27:17.212 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CT.ko 2024-03-17T23:27:17.212 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_DSCP.ko 2024-03-17T23:27:17.213 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_HL.ko 2024-03-17T23:27:17.213 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_HMARK.ko 2024-03-17T23:27:17.213 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-17T23:27:17.213 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_LED.ko 2024-03-17T23:27:17.213 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_LOG.ko 2024-03-17T23:27:17.213 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-17T23:27:17.213 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NETMAP.ko 2024-03-17T23:27:17.213 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NFLOG.ko 2024-03-17T23:27:17.213 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-17T23:27:17.213 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_RATEEST.ko 2024-03-17T23:27:17.213 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-17T23:27:17.213 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_SECMARK.ko 2024-03-17T23:27:17.213 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-17T23:27:17.214 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-17T23:27:17.214 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TEE.ko 2024-03-17T23:27:17.214 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TPROXY.ko 2024-03-17T23:27:17.214 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TRACE.ko 2024-03-17T23:27:17.214 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_addrtype.ko 2024-03-17T23:27:17.214 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_bpf.ko 2024-03-17T23:27:17.214 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cgroup.ko 2024-03-17T23:27:17.214 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cluster.ko 2024-03-17T23:27:17.214 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_comment.ko 2024-03-17T23:27:17.214 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connbytes.ko 2024-03-17T23:27:17.214 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connlabel.ko 2024-03-17T23:27:17.214 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connlimit.ko 2024-03-17T23:27:17.214 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connmark.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_conntrack.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cpu.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_dccp.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_devgroup.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_dscp.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ecn.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_esp.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_hashlimit.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_helper.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_hl.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ipcomp.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_iprange.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ipvs.ko 2024-03-17T23:27:17.215 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_l2tp.ko 2024-03-17T23:27:17.216 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_length.ko 2024-03-17T23:27:17.216 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_limit.ko 2024-03-17T23:27:17.216 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_mac.ko 2024-03-17T23:27:17.216 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_mark.ko 2024-03-17T23:27:17.216 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_multiport.ko 2024-03-17T23:27:17.216 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_nat.ko 2024-03-17T23:27:17.216 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_nfacct.ko 2024-03-17T23:27:17.216 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_osf.ko 2024-03-17T23:27:17.216 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_owner.ko 2024-03-17T23:27:17.216 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_physdev.ko 2024-03-17T23:27:17.216 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_pkttype.ko 2024-03-17T23:27:17.216 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_policy.ko 2024-03-17T23:27:17.216 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_quota.ko 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_rateest.ko 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_realm.ko 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_recent.ko 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_sctp.ko 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_set.ko 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_socket.ko 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_state.ko 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_statistic.ko 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_string.ko 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_tcpmss.ko 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_time.ko 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_u32.ko 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netlink 2024-03-17T23:27:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netlink/netlink_diag.ko 2024-03-17T23:27:17.218 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/nsh 2024-03-17T23:27:17.218 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/nsh/nsh.ko 2024-03-17T23:27:17.218 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch 2024-03-17T23:27:17.218 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/openvswitch.ko 2024-03-17T23:27:17.218 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-geneve.ko 2024-03-17T23:27:17.218 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-gre.ko 2024-03-17T23:27:17.218 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-vxlan.ko 2024-03-17T23:27:17.218 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/packet 2024-03-17T23:27:17.218 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/packet/af_packet_diag.ko 2024-03-17T23:27:17.218 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds 2024-03-17T23:27:17.218 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds.ko 2024-03-17T23:27:17.218 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds_rdma.ko 2024-03-17T23:27:17.218 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds_tcp.ko 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_csum.ko 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_gact.ko 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_mirred.ko 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_nat.ko 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_pedit.ko 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_police.ko 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_simple.ko 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_skbedit.ko 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_vlan.ko 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_basic.ko 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_bpf.ko 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_flow.ko 2024-03-17T23:27:17.219 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_fw.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_route.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_u32.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_cmp.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_ipset.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_meta.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_nbyte.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_text.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_u32.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_choke.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_codel.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_drr.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_fq.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_gred.ko 2024-03-17T23:27:17.220 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_hfsc.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_hhf.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_htb.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_ingress.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_mqprio.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_mqprio_lib.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_multiq.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_netem.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_pie.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_plug.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_prio.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_qfq.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_red.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_sfb.ko 2024-03-17T23:27:17.221 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_sfq.ko 2024-03-17T23:27:17.222 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_tbf.ko 2024-03-17T23:27:17.222 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_teql.ko 2024-03-17T23:27:17.222 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp 2024-03-17T23:27:17.222 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp/sctp.ko 2024-03-17T23:27:17.222 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp/sctp_diag.ko 2024-03-17T23:27:17.222 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc 2024-03-17T23:27:17.222 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss 2024-03-17T23:27:17.222 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-17T23:27:17.222 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-17T23:27:17.222 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/sunrpc.ko 2024-03-17T23:27:17.222 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/xprtrdma 2024-03-17T23:27:17.222 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-17T23:27:17.222 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc/diag.ko 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc/tipc.ko 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/unix 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/unix/unix_diag.ko 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock.ko 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/xfrm 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys 2024-03-17T23:27:17.223 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/encrypted-keys 2024-03-17T23:27:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-17T23:27:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/trusted-keys 2024-03-17T23:27:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/trusted-keys/trusted.ko 2024-03-17T23:27:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt 2024-03-17T23:27:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt/lib 2024-03-17T23:27:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt/lib/irqbypass.ko 2024-03-17T23:27:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.alias 2024-03-17T23:27:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.alias.bin 2024-03-17T23:27:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin 2024-03-17T23:27:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.alias.bin 2024-03-17T23:27:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.bin 2024-03-17T23:27:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.modinfo 2024-03-17T23:27:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.dep 2024-03-17T23:27:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.dep.bin 2024-03-17T23:27:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.devname 2024-03-17T23:27:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.order 2024-03-17T23:27:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.softdep 2024-03-17T23:27:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.symbols 2024-03-17T23:27:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.symbols.bin 2024-03-17T23:27:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/vmlinuz 2024-03-17T23:27:17.229 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc7-g3845b7ac715a 2024-03-17T23:27:17.229 DEBUG:teuthology.orchestra.run.smithi012:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-17T23:27:17.257 INFO:teuthology.orchestra.run.smithi012.stdout:postinstall scriptlet (using /bin/sh): 2024-03-17T23:27:17.258 INFO:teuthology.orchestra.run.smithi012.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-17T23:27:17.258 INFO:teuthology.orchestra.run.smithi012.stdout: /usr/bin/kernel-install add 6.8.0-rc7-g3845b7ac715a /lib/modules/6.8.0-rc7-g3845b7ac715a/vmlinuz 2024-03-17T23:27:17.258 INFO:teuthology.orchestra.run.smithi012.stdout:fi 2024-03-17T23:27:17.258 INFO:teuthology.orchestra.run.smithi012.stdout:for file in vmlinuz System.map config; do 2024-03-17T23:27:17.258 INFO:teuthology.orchestra.run.smithi012.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc7-g3845b7ac715a/${file}" "/boot/${file}-6.8.0-rc7-g3845b7ac715a"; then 2024-03-17T23:27:17.258 INFO:teuthology.orchestra.run.smithi012.stdout: cp "/lib/modules/6.8.0-rc7-g3845b7ac715a/${file}" "/boot/${file}-6.8.0-rc7-g3845b7ac715a" 2024-03-17T23:27:17.258 INFO:teuthology.orchestra.run.smithi012.stdout: fi 2024-03-17T23:27:17.258 INFO:teuthology.orchestra.run.smithi012.stdout:done 2024-03-17T23:27:17.258 INFO:teuthology.orchestra.run.smithi012.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-17T23:27:17.258 INFO:teuthology.orchestra.run.smithi012.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-17T23:27:17.258 INFO:teuthology.orchestra.run.smithi012.stdout:new-kernel-pkg --remove 6.8.0-rc7-g3845b7ac715a --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc7-g3845b7ac715a.img 2024-03-17T23:27:17.258 INFO:teuthology.orchestra.run.smithi012.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-17T23:27:17.259 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-install remove 6.8.0-rc7-g3845b7ac715a 2024-03-17T23:27:17.259 INFO:teuthology.orchestra.run.smithi012.stdout:fi 2024-03-17T23:27:17.259 INFO:teuthology.orchestra.run.smithi012.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-17T23:27:17.259 INFO:teuthology.orchestra.run.smithi012.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-17T23:27:17.259 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/update-bootloader --remove 6.8.0-rc7-g3845b7ac715a 2024-03-17T23:27:17.259 INFO:teuthology.orchestra.run.smithi012.stdout:fi 2024-03-17T23:27:17.260 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2024-03-17T23:27:17.343 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2024-03-17T23:27:17.343 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2024-03-17T23:27:17.343 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.06 2024-03-17T23:27:17.343 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 68.el9 2024-03-17T23:27:17.343 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2024-03-17T23:27:17.343 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-17T23:27:17.343 INFO:teuthology.orchestra.run.smithi012.stdout:Group : Unspecified 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 8264254 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : builder@centos.org 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : CentOS 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-17T23:27:17.344 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-17T23:27:17.345 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-17T23:27:17.345 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2024-03-17T23:27:17.345 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:27:17.345 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2024-03-17T23:27:17.346 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-17T23:27:17.346 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 6.8.0-rc7-g3845b7ac715a 2024-03-17T23:27:17.346 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-17T23:27:17.973 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2024-03-17T23:27:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-17T23:27:19.346 INFO:teuthology.orchestra.run.smithi012.stderr:done 2024-03-17T23:27:19.348 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-03-17T23:27:19.363 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.SDURy61hI8 2024-03-17T23:27:19.364 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SDURy61hI8 2024-03-17T23:27:19.441 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.SDURy61hI8 2024-03-17T23:27:19.586 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.SDURy61hI8 is 6KB 2024-03-17T23:27:19.606 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.SDURy61hI8 2024-03-17T23:27:19.621 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2024-03-17T23:27:19.695 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-17T23:27:19.695 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-17T23:27:19.695 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-17T23:27:19.695 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc7-g3845b7ac715a.conf 2024-03-17T23:27:19.696 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc7-g3845b7ac715a 2024-03-17T23:27:19.863 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2024-03-17T23:27:49.866 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-17T23:27:49.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-17T23:27:49.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:28:08.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-03-17T23:28:17.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-17T23:28:17.225 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:28:35.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-03-17T23:28:47.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-17T23:28:47.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:28:50.691 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-03-17T23:29:05.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-17T23:29:05.693 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:29:06.144 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-03-17T23:29:06.783 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-17T23:29:06.783 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc7-g3845b7ac715a"... 2024-03-17T23:29:06.784 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-03-17T23:29:06.830 INFO:teuthology.orchestra.run.smithi012.stdout:6.8.0-rc7-g3845b7ac715a 2024-03-17T23:29:06.830 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc7-g3845b7ac715a vs 6.8.0-rc7-g3845b7ac715a 2024-03-17T23:29:06.830 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-17T23:29:06.830 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-17T23:29:07.831 DEBUG:teuthology.parallel:result is None 2024-03-17T23:29:07.831 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-17T23:29:07.842 INFO:teuthology.task.internal:Creating test directory... 2024-03-17T23:29:07.842 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-17T23:29:07.847 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-17T23:29:07.850 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-17T23:29:07.885 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-17T23:29:07.891 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-17T23:29:07.900 INFO:teuthology.task.internal:Creating archive directory... 2024-03-17T23:29:07.901 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-17T23:29:07.935 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-17T23:29:07.937 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-17T23:29:08.070 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-17T23:29:08.077 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-17T23:29:08.077 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-17T23:29:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:29:08.094 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-17T23:29:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:29:08.112 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-17T23:29:08.119 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-17T23:29:08.137 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-17T23:29:08.173 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-17T23:29:08.194 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-17T23:29:08.207 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-17T23:29:08.226 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-17T23:29:08.320 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-17T23:29:08.409 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-17T23:29:08.420 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-17T23:29:08.427 INFO:teuthology.task.internal:Configuring sudo... 2024-03-17T23:29:08.428 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-17T23:29:08.439 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-17T23:29:08.441 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-17T23:29:08.570 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-17T23:29:08.592 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-17T23:29:08.593 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-17T23:29:08.602 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-17T23:29:08.604 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-17T23:29:08.673 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-17T23:29:08.789 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-17T23:29:08.934 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-17T23:29:09.040 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-17T23:29:09.176 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:29:09.176 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-17T23:29:09.313 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-17T23:29:09.360 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-17T23:29:09.408 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-17T23:29:09.492 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-17T23:29:09.536 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-17T23:29:09.536 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-17T23:29:09.606 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-17T23:29:09.649 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-17T23:29:09.829 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-17T23:29:09.874 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-17T23:29:09.919 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-17T23:29:09.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-17T23:29:09.990 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2024-03-17T23:29:09.997 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-03-17T23:29:09.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-03-17T23:29:10.038 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-17T23:29:10.072 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-17T23:29:10.104 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-17T23:29:10.501 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-17T23:29:10.509 INFO:teuthology.task.internal:Starting timer... 2024-03-17T23:29:10.509 INFO:teuthology.run_tasks:Running task pcp... 2024-03-17T23:29:10.522 INFO:teuthology.run_tasks:Running task selinux... 2024-03-17T23:29:10.532 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-17T23:29:10.532 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2024-03-17T23:29:10.642 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: 2024-03-17T23:29:10.646 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2024-03-17T23:29:10.706 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2024-03-17T23:29:10.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-03-17T23:29:10.766 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-03-17T23:29:10.767 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-17T23:29:10.768 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2024-03-17T23:29:10.820 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2024-03-17T23:29:10.820 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2024-03-17T23:29:10.850 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2024-03-17T23:29:10.850 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-03-17T23:29:10.875 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-03-17T23:29:10.875 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi079.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-03-17T23:29:10.875 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-17T23:29:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:29:10.960 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-17T23:29:10.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:29:10.989 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-17T23:29:11.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:29:11.016 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-17T23:29:11.016 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2024-03-17T23:29:11.108 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2024-03-17T23:29:11.142 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-03-17T23:29:11.175 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-17T23:29:11.185 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-17T23:29:11.202 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-17T23:29:11.204 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi079.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-17T23:35:17.919 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-03-17T23:35:17.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-17T23:35:17.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:35:18.077 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-03-17T23:35:18.238 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-17T23:35:18.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-17T23:35:18.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:35:18.335 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-03-17T23:35:18.401 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-17T23:35:18.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-17T23:35:18.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:35:18.502 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-03-17T23:35:18.575 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-17T23:35:18.575 INFO:teuthology.run_tasks:Running task clock... 2024-03-17T23:35:18.589 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-17T23:35:18.589 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-17T23:35:18.589 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-17T23:35:18.603 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-17T23:35:18.604 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-17T23:35:18.606 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-17T23:35:18.606 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-17T23:35:18.633 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-17T23:35:18.649 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-17T23:35:18.650 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-17T23:35:18.667 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-17T23:35:18.678 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2024-03-17T23:35:18.692 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2024-03-17T23:35:18.698 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-17T23:35:18.700 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-03-17T23:35:18.708 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-17T23:35:18.715 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-03-17T23:35:18.726 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-17T23:35:18.733 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-17T23:35:18.749 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-17T23:35:18.753 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-17T23:35:18.767 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-03-17T23:35:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-17T23:35:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-03-17T23:35:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:35:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:35:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:35:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:35:18.793 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-17T23:35:18.796 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-17T23:35:18.796 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-17T23:35:18.796 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:35:18.796 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:35:18.796 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:35:18.796 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:35:18.852 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: ntpd: command not found 2024-03-17T23:35:18.897 INFO:teuthology.orchestra.run.smithi012.stdout:506 Cannot talk to daemon 2024-03-17T23:35:18.952 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-17T23:35:19.006 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-17T23:35:19.147 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-03-17T23:35:19.155 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-17T23:35:19.155 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-03-17T23:35:19.155 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:35:19.155 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:35:19.155 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:35:19.156 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:35:19.157 INFO:teuthology.run_tasks:Running task install... 2024-03-17T23:35:19.166 DEBUG:teuthology.task.install:project ceph 2024-03-17T23:35:19.166 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-17T23:35:19.166 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-17T23:35:19.166 INFO:teuthology.task.install:Using flavor: default 2024-03-17T23:35:19.173 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-17T23:35:19.173 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-17T23:35:19.174 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-17T23:35:19.174 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:35:19.176 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-17T23:35:19.176 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:35:19.177 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-17T23:35:19.178 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:35:19.324 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-17T23:35:19.324 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-17T23:35:19.398 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-17T23:35:19.398 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-17T23:35:19.422 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-17T23:35:19.422 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-17T23:35:19.442 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-17T23:35:19.443 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-17T23:35:19.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-17T23:35:19.472 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-17T23:35:19.472 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-17T23:35:19.520 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-17T23:35:19.520 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:35:19.520 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-17T23:35:19.541 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-17T23:35:19.541 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-17T23:35:19.541 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-17T23:35:19.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-17T23:35:19.574 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-17T23:35:19.575 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-17T23:35:19.626 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-17T23:35:19.627 DEBUG:teuthology.orchestra.run.smithi012:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-17T23:35:19.628 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-17T23:35:19.643 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-17T23:35:19.692 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2024-03-17T23:35:19.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2024-03-17T23:35:19.725 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-17T23:35:19.771 DEBUG:teuthology.orchestra.run.smithi012:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-17T23:35:19.792 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2024-03-17T23:35:19.793 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2024-03-17T23:35:19.913 DEBUG:teuthology.orchestra.run.smithi012:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-17T23:35:19.931 INFO:teuthology.orchestra.run.smithi191.stdout:74 files removed 2024-03-17T23:35:19.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-17T23:35:20.012 INFO:teuthology.orchestra.run.smithi012.stdout:check_obsoletes = 1 2024-03-17T23:35:20.017 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum clean all 2024-03-17T23:35:20.032 INFO:teuthology.orchestra.run.smithi079.stdout:74 files removed 2024-03-17T23:35:20.059 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-17T23:35:20.389 INFO:teuthology.orchestra.run.smithi012.stdout:74 files removed 2024-03-17T23:35:20.429 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-17T23:35:20.774 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 158 kB/s | 83 kB 00:00 2024-03-17T23:35:20.854 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 164 kB/s | 83 kB 00:00 2024-03-17T23:35:21.100 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 35 kB/s | 10 kB 00:00 2024-03-17T23:35:21.329 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-17T23:35:21.488 INFO:teuthology.orchestra.run.smithi012.stdout:ceph packages for x86_64 153 kB/s | 83 kB 00:00 2024-03-17T23:35:21.609 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-03-17T23:35:21.813 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2024-03-17T23:35:21.972 INFO:teuthology.orchestra.run.smithi012.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-17T23:35:22.004 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-17T23:35:22.218 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-17T23:35:22.540 INFO:teuthology.orchestra.run.smithi012.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-03-17T23:35:23.193 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-17T23:35:23.420 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-17T23:35:23.657 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-17T23:35:24.503 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-17T23:35:24.734 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-17T23:35:24.860 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-17T23:35:25.898 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-17T23:35:28.737 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-17T23:35:29.413 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 19 MB 00:03 2024-03-17T23:35:29.490 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 5.1 MB/s | 19 MB 00:03 2024-03-17T23:35:34.323 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.1 MB 00:01 2024-03-17T23:35:35.848 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 16 kB 00:00 2024-03-17T23:35:37.626 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-17T23:35:42.551 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-17T23:35:45.014 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-17T23:35:45.015 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-17T23:35:45.029 INFO:teuthology.orchestra.run.smithi191.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-17T23:35:45.031 INFO:teuthology.orchestra.run.smithi191.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-17T23:35:45.032 INFO:teuthology.orchestra.run.smithi191.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-17T23:35:45.032 INFO:teuthology.orchestra.run.smithi191.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-17T23:35:45.034 INFO:teuthology.orchestra.run.smithi191.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-17T23:35:45.037 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-17T23:35:45.038 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-17T23:35:45.117 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-17T23:35:45.137 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-17T23:35:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-17T23:35:45.138 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-17T23:35:45.138 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-17T23:35:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-17T23:35:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-17T23:35:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-17T23:35:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-17T23:35:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-17T23:35:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-17T23:35:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-17T23:35:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-17T23:35:45.138 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-17T23:35:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-17T23:35:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-17T23:35:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-17T23:35:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-17T23:35:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-17T23:35:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-17T23:35:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-17T23:35:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-17T23:35:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-17T23:35:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-17T23:35:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-17T23:35:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-17T23:35:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-17T23:35:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-17T23:35:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-17T23:35:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-17T23:35:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-17T23:35:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-17T23:35:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-17T23:35:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-17T23:35:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-17T23:35:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-17T23:35:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-17T23:35:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-17T23:35:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-17T23:35:45.140 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-17T23:35:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-17T23:35:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-17T23:35:45.141 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2024-03-17T23:35:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-17T23:35:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-17T23:35:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-17T23:35:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-17T23:35:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-17T23:35:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-17T23:35:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-17T23:35:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-17T23:35:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-17T23:35:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-17T23:35:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-17T23:35:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-17T23:35:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-17T23:35:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-17T23:35:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-17T23:35:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-17T23:35:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-17T23:35:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-17T23:35:45.142 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-17T23:35:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-17T23:35:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-17T23:35:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-17T23:35:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-17T23:35:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-17T23:35:45.143 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-17T23:35:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-17T23:35:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-17T23:35:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-17T23:35:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-17T23:35:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-17T23:35:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-17T23:35:45.143 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-17T23:35:45.144 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-17T23:35:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-17T23:35:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-17T23:35:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-17T23:35:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-17T23:35:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-17T23:35:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-17T23:35:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-17T23:35:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-17T23:35:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-17T23:35:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-17T23:35:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-17T23:35:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-17T23:35:45.145 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-17T23:35:45.146 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-17T23:35:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-17T23:35:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-17T23:35:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-17T23:35:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-17T23:35:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-17T23:35:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-17T23:35:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-17T23:35:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-17T23:35:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-17T23:35:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-17T23:35:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-17T23:35:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-17T23:35:45.147 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-17T23:35:45.148 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-17T23:35:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-17T23:35:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-17T23:35:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-17T23:35:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-17T23:35:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-17T23:35:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-17T23:35:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-17T23:35:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-17T23:35:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-17T23:35:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-17T23:35:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-17T23:35:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-17T23:35:45.149 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-17T23:35:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-17T23:35:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-17T23:35:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-17T23:35:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-17T23:35:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-17T23:35:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-17T23:35:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-17T23:35:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-17T23:35:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-17T23:35:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-17T23:35:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-17T23:35:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-17T23:35:45.150 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-17T23:35:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-17T23:35:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-17T23:35:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-17T23:35:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-17T23:35:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-17T23:35:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-17T23:35:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-17T23:35:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-17T23:35:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-17T23:35:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-17T23:35:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-17T23:35:45.151 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2024-03-17T23:35:45.151 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-17T23:35:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:35:45.152 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-17T23:35:45.152 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-17T23:35:45.152 INFO:teuthology.orchestra.run.smithi191.stdout:Install 150 Packages 2024-03-17T23:35:45.152 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 24 Packages 2024-03-17T23:35:45.152 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:35:45.152 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 224 M 2024-03-17T23:35:45.153 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-17T23:35:45.952 INFO:teuthology.orchestra.run.smithi191.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 35 kB/s | 6.6 kB 00:00 2024-03-17T23:35:53.294 INFO:teuthology.orchestra.run.smithi191.stdout:(2/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 728 kB/s | 5.4 MB 00:07 2024-03-17T23:35:53.328 INFO:teuthology.orchestra.run.smithi191.stdout:(3/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 121 kB/s | 890 kB 00:07 2024-03-17T23:35:53.355 INFO:teuthology.orchestra.run.smithi191.stdout:(4/174): ceph-immutable-object-cache-19.0.0-216 2.4 MB/s | 145 kB 00:00 2024-03-17T23:35:53.662 INFO:teuthology.orchestra.run.smithi191.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 4.8 MB/s | 1.5 MB 00:00 2024-03-17T23:35:54.078 INFO:teuthology.orchestra.run.smithi191.stdout:(6/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 2.5 MB/s | 21 MB 00:08 2024-03-17T23:35:54.270 INFO:teuthology.orchestra.run.smithi191.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 7.7 MB/s | 4.7 MB 00:00 2024-03-17T23:35:54.538 INFO:teuthology.orchestra.run.smithi191.stdout:(8/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 2.0 MB/s | 2.4 MB 00:01 2024-03-17T23:35:54.638 INFO:teuthology.orchestra.run.smithi191.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 252 kB/s | 25 kB 00:00 2024-03-17T23:35:56.080 INFO:teuthology.orchestra.run.smithi191.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 5.4 MB/s | 9.8 MB 00:01 2024-03-17T23:35:56.189 INFO:teuthology.orchestra.run.smithi191.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.0 MB/s | 221 kB 00:00 2024-03-17T23:35:56.382 INFO:teuthology.orchestra.run.smithi191.stdout:(12/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 7.3 MB/s | 17 MB 00:02 2024-03-17T23:35:56.406 INFO:teuthology.orchestra.run.smithi191.stdout:(13/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 125 kB/s | 27 kB 00:00 2024-03-17T23:35:56.465 INFO:teuthology.orchestra.run.smithi191.stdout:(14/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 380 kB/s | 32 kB 00:00 2024-03-17T23:35:56.565 INFO:teuthology.orchestra.run.smithi191.stdout:(15/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 4.6 MB/s | 747 kB 00:00 2024-03-17T23:35:56.590 INFO:teuthology.orchestra.run.smithi191.stdout:(16/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.3 MB/s | 163 kB 00:00 2024-03-17T23:35:56.658 INFO:teuthology.orchestra.run.smithi191.stdout:(17/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.3 MB/s | 127 kB 00:00 2024-03-17T23:35:56.750 INFO:teuthology.orchestra.run.smithi191.stdout:(18/174): libradosstriper1-19.0.0-2162.g3d78f5b 3.1 MB/s | 501 kB 00:00 2024-03-17T23:35:56.875 INFO:teuthology.orchestra.run.smithi191.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 361 kB/s | 45 kB 00:00 2024-03-17T23:35:56.975 INFO:teuthology.orchestra.run.smithi191.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.3 MB/s | 131 kB 00:00 2024-03-17T23:35:57.084 INFO:teuthology.orchestra.run.smithi191.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.5 MB/s | 162 kB 00:00 2024-03-17T23:35:57.201 INFO:teuthology.orchestra.run.smithi191.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 2.7 MB/s | 320 kB 00:00 2024-03-17T23:35:57.344 INFO:teuthology.orchestra.run.smithi191.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.1 MB/s | 298 kB 00:00 2024-03-17T23:35:57.452 INFO:teuthology.orchestra.run.smithi191.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 918 kB/s | 99 kB 00:00 2024-03-17T23:35:57.565 INFO:teuthology.orchestra.run.smithi191.stdout:(25/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 5.7 MB/s | 5.1 MB 00:00 2024-03-17T23:35:57.589 INFO:teuthology.orchestra.run.smithi191.stdout:(26/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 638 kB/s | 85 kB 00:00 2024-03-17T23:35:57.703 INFO:teuthology.orchestra.run.smithi191.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.4 MB/s | 170 kB 00:00 2024-03-17T23:35:57.803 INFO:teuthology.orchestra.run.smithi191.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 339 kB/s | 34 kB 00:00 2024-03-17T23:35:57.904 INFO:teuthology.orchestra.run.smithi191.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.4 MB/s | 139 kB 00:00 2024-03-17T23:35:58.279 INFO:teuthology.orchestra.run.smithi191.stdout:(30/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 4.3 MB/s | 3.1 MB 00:00 2024-03-17T23:35:58.321 INFO:teuthology.orchestra.run.smithi191.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 4.3 MB/s | 1.8 MB 00:00 2024-03-17T23:35:58.780 INFO:teuthology.orchestra.run.smithi191.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 542 kB/s | 248 kB 00:00 2024-03-17T23:35:58.880 INFO:teuthology.orchestra.run.smithi191.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 492 kB/s | 49 kB 00:00 2024-03-17T23:35:58.989 INFO:teuthology.orchestra.run.smithi191.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 152 kB/s | 16 kB 00:00 2024-03-17T23:35:59.073 INFO:teuthology.orchestra.run.smithi191.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 3.1 MB/s | 264 kB 00:00 2024-03-17T23:35:59.390 INFO:teuthology.orchestra.run.smithi191.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 2.3 MB/s | 748 kB 00:00 2024-03-17T23:35:59.465 INFO:teuthology.orchestra.run.smithi191.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 243 kB/s | 18 kB 00:00 2024-03-17T23:35:59.599 INFO:teuthology.orchestra.run.smithi191.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.9 MB/s | 662 kB 00:00 2024-03-17T23:35:59.658 INFO:teuthology.orchestra.run.smithi191.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-17T23:35:59.692 INFO:teuthology.orchestra.run.smithi191.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-17T23:35:59.734 INFO:teuthology.orchestra.run.smithi191.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-17T23:35:59.768 INFO:teuthology.orchestra.run.smithi191.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 924 kB/s | 32 kB 00:00 2024-03-17T23:35:59.861 INFO:teuthology.orchestra.run.smithi191.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-17T23:35:59.890 INFO:teuthology.orchestra.run.smithi191.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 690 kB/s | 17 kB 00:00 2024-03-17T23:35:59.919 INFO:teuthology.orchestra.run.smithi191.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 303 kB/s | 10 kB 00:00 2024-03-17T23:35:59.953 INFO:teuthology.orchestra.run.smithi191.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.6 MB/s | 159 kB 00:00 2024-03-17T23:35:59.979 INFO:teuthology.orchestra.run.smithi191.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-17T23:36:00.029 INFO:teuthology.orchestra.run.smithi191.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-17T23:36:00.079 INFO:teuthology.orchestra.run.smithi191.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-17T23:36:00.113 INFO:teuthology.orchestra.run.smithi191.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-17T23:36:00.155 INFO:teuthology.orchestra.run.smithi191.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-17T23:36:00.197 INFO:teuthology.orchestra.run.smithi191.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-17T23:36:00.289 INFO:teuthology.orchestra.run.smithi191.stdout:(53/174): ceph-mgr-diskprediction-local-19.0.0- 3.7 MB/s | 7.4 MB 00:02 2024-03-17T23:36:00.314 INFO:teuthology.orchestra.run.smithi191.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2024-03-17T23:36:00.339 INFO:teuthology.orchestra.run.smithi191.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 690 kB/s | 34 kB 00:00 2024-03-17T23:36:00.439 INFO:teuthology.orchestra.run.smithi191.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-17T23:36:00.490 INFO:teuthology.orchestra.run.smithi191.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 16 MB/s | 811 kB 00:00 2024-03-17T23:36:00.532 INFO:teuthology.orchestra.run.smithi191.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 8.0 MB/s | 1.5 MB 00:00 2024-03-17T23:36:00.557 INFO:teuthology.orchestra.run.smithi191.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-17T23:36:00.674 INFO:teuthology.orchestra.run.smithi191.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-17T23:36:00.708 INFO:teuthology.orchestra.run.smithi191.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.8 MB/s | 1.0 MB 00:00 2024-03-17T23:36:00.725 INFO:teuthology.orchestra.run.smithi191.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.6 MB/s | 78 kB 00:00 2024-03-17T23:36:00.800 INFO:teuthology.orchestra.run.smithi191.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 328 kB/s | 24 kB 00:00 2024-03-17T23:36:00.867 INFO:teuthology.orchestra.run.smithi191.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 36 MB/s | 5.8 MB 00:00 2024-03-17T23:36:00.892 INFO:teuthology.orchestra.run.smithi191.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 197 kB/s | 18 kB 00:00 2024-03-17T23:36:00.918 INFO:teuthology.orchestra.run.smithi191.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-17T23:36:00.942 INFO:teuthology.orchestra.run.smithi191.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 634 kB/s | 32 kB 00:00 2024-03-17T23:36:01.043 INFO:teuthology.orchestra.run.smithi191.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-17T23:36:01.068 INFO:teuthology.orchestra.run.smithi191.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-17T23:36:01.093 INFO:teuthology.orchestra.run.smithi191.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-17T23:36:01.127 INFO:teuthology.orchestra.run.smithi191.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 69 kB 00:00 2024-03-17T23:36:01.153 INFO:teuthology.orchestra.run.smithi191.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 702 kB/s | 41 kB 00:00 2024-03-17T23:36:01.179 INFO:teuthology.orchestra.run.smithi191.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-17T23:36:01.203 INFO:teuthology.orchestra.run.smithi191.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-17T23:36:01.229 INFO:teuthology.orchestra.run.smithi191.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-17T23:36:01.254 INFO:teuthology.orchestra.run.smithi191.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-03-17T23:36:01.279 INFO:teuthology.orchestra.run.smithi191.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 854 kB/s | 43 kB 00:00 2024-03-17T23:36:01.388 INFO:teuthology.orchestra.run.smithi191.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 366 kB/s | 39 kB 00:00 2024-03-17T23:36:01.588 INFO:teuthology.orchestra.run.smithi191.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-17T23:36:01.618 INFO:teuthology.orchestra.run.smithi191.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 184 kB/s | 41 kB 00:00 2024-03-17T23:36:01.638 INFO:teuthology.orchestra.run.smithi191.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-17T23:36:01.664 INFO:teuthology.orchestra.run.smithi191.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-17T23:36:01.713 INFO:teuthology.orchestra.run.smithi191.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 855 kB/s | 63 kB 00:00 2024-03-17T23:36:01.739 INFO:teuthology.orchestra.run.smithi191.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-03-17T23:36:01.765 INFO:teuthology.orchestra.run.smithi191.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-17T23:36:01.790 INFO:teuthology.orchestra.run.smithi191.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-17T23:36:01.823 INFO:teuthology.orchestra.run.smithi191.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-17T23:36:01.848 INFO:teuthology.orchestra.run.smithi191.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-17T23:36:01.874 INFO:teuthology.orchestra.run.smithi191.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-17T23:36:01.899 INFO:teuthology.orchestra.run.smithi191.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-17T23:36:01.975 INFO:teuthology.orchestra.run.smithi191.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2024-03-17T23:36:02.025 INFO:teuthology.orchestra.run.smithi191.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 6.3 MB/s | 803 kB 00:00 2024-03-17T23:36:02.084 INFO:teuthology.orchestra.run.smithi191.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 195 kB 00:00 2024-03-17T23:36:02.109 INFO:teuthology.orchestra.run.smithi191.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.4 MB/s | 125 kB 00:00 2024-03-17T23:36:02.426 INFO:teuthology.orchestra.run.smithi191.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 63 kB/s | 21 kB 00:00 2024-03-17T23:36:02.451 INFO:teuthology.orchestra.run.smithi191.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 74 kB/s | 25 kB 00:00 2024-03-17T23:36:02.905 INFO:teuthology.orchestra.run.smithi191.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 545 kB/s | 245 kB 00:00 2024-03-17T23:36:03.152 INFO:teuthology.orchestra.run.smithi191.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 992 kB/s | 248 kB 00:00 2024-03-17T23:36:03.260 INFO:teuthology.orchestra.run.smithi191.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.0 MB/s | 216 kB 00:00 2024-03-17T23:36:03.378 INFO:teuthology.orchestra.run.smithi191.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 299 kB/s | 35 kB 00:00 2024-03-17T23:36:03.503 INFO:teuthology.orchestra.run.smithi191.stdout:(101/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.8 MB/s | 4.1 MB 00:01 2024-03-17T23:36:03.528 INFO:teuthology.orchestra.run.smithi191.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 296 kB/s | 44 kB 00:00 2024-03-17T23:36:03.620 INFO:teuthology.orchestra.run.smithi191.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 800 kB/s | 94 kB 00:00 2024-03-17T23:36:03.804 INFO:teuthology.orchestra.run.smithi191.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 81 kB/s | 22 kB 00:00 2024-03-17T23:36:04.187 INFO:teuthology.orchestra.run.smithi191.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-03-17T23:36:04.262 INFO:teuthology.orchestra.run.smithi191.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 4.0 MB/s | 309 kB 00:00 2024-03-17T23:36:04.329 INFO:teuthology.orchestra.run.smithi191.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-03-17T23:36:04.672 INFO:teuthology.orchestra.run.smithi191.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-03-17T23:36:04.730 INFO:teuthology.orchestra.run.smithi191.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 447 kB/s | 26 kB 00:00 2024-03-17T23:36:04.788 INFO:teuthology.orchestra.run.smithi191.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 836 kB/s | 49 kB 00:00 2024-03-17T23:36:04.847 INFO:teuthology.orchestra.run.smithi191.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-17T23:36:04.906 INFO:teuthology.orchestra.run.smithi191.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-03-17T23:36:05.023 INFO:teuthology.orchestra.run.smithi191.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-17T23:36:05.098 INFO:teuthology.orchestra.run.smithi191.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-03-17T23:36:05.165 INFO:teuthology.orchestra.run.smithi191.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 655 kB/s | 43 kB 00:00 2024-03-17T23:36:05.224 INFO:teuthology.orchestra.run.smithi191.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 551 kB/s | 32 kB 00:00 2024-03-17T23:36:05.282 INFO:teuthology.orchestra.run.smithi191.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 258 kB/s | 15 kB 00:00 2024-03-17T23:36:05.341 INFO:teuthology.orchestra.run.smithi191.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-17T23:36:05.408 INFO:teuthology.orchestra.run.smithi191.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 5.3 MB/s | 358 kB 00:00 2024-03-17T23:36:05.501 INFO:teuthology.orchestra.run.smithi191.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-17T23:36:05.567 INFO:teuthology.orchestra.run.smithi191.stdout:(121/174): protobuf-compiler-3.14.0-13.el9.x86_ 444 kB/s | 863 kB 00:01 2024-03-17T23:36:05.592 INFO:teuthology.orchestra.run.smithi191.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 391 kB/s | 35 kB 00:00 2024-03-17T23:36:05.776 INFO:teuthology.orchestra.run.smithi191.stdout:(123/174): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-03-17T23:36:05.842 INFO:teuthology.orchestra.run.smithi191.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-17T23:36:05.868 INFO:teuthology.orchestra.run.smithi191.stdout:(125/174): python3-google-auth-2.28.0-1.el9.noa 674 kB/s | 203 kB 00:00 2024-03-17T23:36:05.893 INFO:teuthology.orchestra.run.smithi191.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-03-17T23:36:05.918 INFO:teuthology.orchestra.run.smithi191.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-03-17T23:36:05.943 INFO:teuthology.orchestra.run.smithi191.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-03-17T23:36:05.969 INFO:teuthology.orchestra.run.smithi191.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-03-17T23:36:05.994 INFO:teuthology.orchestra.run.smithi191.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-03-17T23:36:06.053 INFO:teuthology.orchestra.run.smithi191.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 789 kB/s | 46 kB 00:00 2024-03-17T23:36:06.120 INFO:teuthology.orchestra.run.smithi191.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-03-17T23:36:06.145 INFO:teuthology.orchestra.run.smithi191.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 859 kB/s | 79 kB 00:00 2024-03-17T23:36:06.179 INFO:teuthology.orchestra.run.smithi191.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 987 kB/s | 58 kB 00:00 2024-03-17T23:36:06.204 INFO:teuthology.orchestra.run.smithi191.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-03-17T23:36:06.246 INFO:teuthology.orchestra.run.smithi191.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-03-17T23:36:06.281 INFO:teuthology.orchestra.run.smithi191.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-17T23:36:06.305 INFO:teuthology.orchestra.run.smithi191.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 484 kB/s | 28 kB 00:00 2024-03-17T23:36:06.338 INFO:teuthology.orchestra.run.smithi191.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 528 kB/s | 31 kB 00:00 2024-03-17T23:36:06.364 INFO:teuthology.orchestra.run.smithi191.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-03-17T23:36:06.406 INFO:teuthology.orchestra.run.smithi191.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 885 kB/s | 59 kB 00:00 2024-03-17T23:36:06.431 INFO:teuthology.orchestra.run.smithi191.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 535 kB/s | 36 kB 00:00 2024-03-17T23:36:06.465 INFO:teuthology.orchestra.run.smithi191.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 871 kB/s | 51 kB 00:00 2024-03-17T23:36:06.498 INFO:teuthology.orchestra.run.smithi191.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-03-17T23:36:06.523 INFO:teuthology.orchestra.run.smithi191.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-17T23:36:06.566 INFO:teuthology.orchestra.run.smithi191.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 6.2 MB/s | 427 kB 00:00 2024-03-17T23:36:06.591 INFO:teuthology.orchestra.run.smithi191.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 298 kB/s | 20 kB 00:00 2024-03-17T23:36:06.624 INFO:teuthology.orchestra.run.smithi191.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-17T23:36:06.733 INFO:teuthology.orchestra.run.smithi191.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-17T23:36:07.733 INFO:teuthology.orchestra.run.smithi191.stdout:(150/174): librados2-19.0.0-2162.g3d78f5ba.el9. 3.1 MB/s | 3.4 MB 00:01 2024-03-17T23:36:07.783 INFO:teuthology.orchestra.run.smithi191.stdout:(151/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 3.0 MB/s | 3.1 MB 00:01 2024-03-17T23:36:07.817 INFO:teuthology.orchestra.run.smithi191.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 3.8 MB/s | 321 kB 00:00 2024-03-17T23:36:08.109 INFO:teuthology.orchestra.run.smithi191.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 6.2 MB/s | 1.8 MB 00:00 2024-03-17T23:36:08.159 INFO:teuthology.orchestra.run.smithi191.stdout:(154/174): NetworkManager-1.46.0-1.el9.x86_64.r 6.1 MB/s | 2.3 MB 00:00 2024-03-17T23:36:08.218 INFO:teuthology.orchestra.run.smithi191.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 391 kB/s | 42 kB 00:00 2024-03-17T23:36:08.302 INFO:teuthology.orchestra.run.smithi191.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.7 MB/s | 248 kB 00:00 2024-03-17T23:36:08.318 INFO:teuthology.orchestra.run.smithi191.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 712 kB/s | 71 kB 00:00 2024-03-17T23:36:08.360 INFO:teuthology.orchestra.run.smithi191.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 622 kB/s | 36 kB 00:00 2024-03-17T23:36:08.402 INFO:teuthology.orchestra.run.smithi191.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.2 MB/s | 192 kB 00:00 2024-03-17T23:36:08.453 INFO:teuthology.orchestra.run.smithi191.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-17T23:36:08.478 INFO:teuthology.orchestra.run.smithi191.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 306 kB/s | 23 kB 00:00 2024-03-17T23:36:08.553 INFO:teuthology.orchestra.run.smithi191.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-03-17T23:36:08.578 INFO:teuthology.orchestra.run.smithi191.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-17T23:36:08.670 INFO:teuthology.orchestra.run.smithi191.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 697 kB/s | 64 kB 00:00 2024-03-17T23:36:08.696 INFO:teuthology.orchestra.run.smithi191.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 954 kB/s | 136 kB 00:00 2024-03-17T23:36:08.771 INFO:teuthology.orchestra.run.smithi191.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 286 kB/s | 29 kB 00:00 2024-03-17T23:36:08.846 INFO:teuthology.orchestra.run.smithi191.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 8.1 MB/s | 1.2 MB 00:00 2024-03-17T23:36:09.038 INFO:teuthology.orchestra.run.smithi191.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 8.1 MB/s | 2.2 MB 00:00 2024-03-17T23:36:09.089 INFO:teuthology.orchestra.run.smithi191.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 9.4 MB/s | 2.3 MB 00:00 2024-03-17T23:36:09.172 INFO:teuthology.orchestra.run.smithi191.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 375 kB/s | 31 kB 00:00 2024-03-17T23:36:09.197 INFO:teuthology.orchestra.run.smithi191.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 2.9 MB/s | 465 kB 00:00 2024-03-17T23:36:09.398 INFO:teuthology.orchestra.run.smithi191.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.8 MB/s | 1.1 MB 00:00 2024-03-17T23:36:09.564 INFO:teuthology.orchestra.run.smithi191.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 203 kB/s | 74 kB 00:00 2024-03-17T23:36:12.739 INFO:teuthology.orchestra.run.smithi191.stdout:(174/174): ceph-test-19.0.0-2162.g3d78f5ba.el9. 2.6 MB/s | 48 MB 00:18 2024-03-17T23:36:12.749 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-17T23:36:12.749 INFO:teuthology.orchestra.run.smithi191.stdout:Total 8.1 MB/s | 224 MB 00:27 2024-03-17T23:36:13.202 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-17T23:36:13.472 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-17T23:36:13.472 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-17T23:36:15.271 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-17T23:36:15.272 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-17T23:36:22.097 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-17T23:36:22.209 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-17T23:36:22.252 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-17T23:36:22.282 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-17T23:36:22.292 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-17T23:36:22.332 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-17T23:36:22.385 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-17T23:36:22.420 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-17T23:36:22.454 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-17T23:36:22.495 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-17T23:36:22.814 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-17T23:36:22.871 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:36:22.992 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:36:23.022 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:36:23.113 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:36:23.142 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-17T23:36:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-17T23:36:23.358 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-17T23:36:23.415 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-17T23:36:23.444 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:36:23.539 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:36:23.562 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-17T23:36:23.609 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-17T23:36:23.648 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-17T23:36:23.671 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-17T23:36:23.687 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-17T23:36:23.879 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-17T23:36:23.910 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-17T23:36:23.942 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-17T23:36:24.047 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-17T23:36:24.130 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-17T23:36:24.188 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-17T23:36:24.249 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:36:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:36:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:36:24.669 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-17T23:36:24.755 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-17T23:36:24.794 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-17T23:36:24.821 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-17T23:36:24.864 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-17T23:36:24.939 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-17T23:36:24.992 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-17T23:36:25.031 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-17T23:36:25.095 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-17T23:36:25.130 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-17T23:36:25.174 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-17T23:36:25.206 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-17T23:36:25.246 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-17T23:36:25.312 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-17T23:36:25.347 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-17T23:36:25.382 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-17T23:36:25.411 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-17T23:36:25.483 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-17T23:36:25.526 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-17T23:36:25.599 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-17T23:36:25.741 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-17T23:36:25.794 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-17T23:36:26.317 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-17T23:36:26.399 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-17T23:36:26.485 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-17T23:36:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-17T23:36:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-17T23:36:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-17T23:36:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-17T23:36:26.637 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-17T23:36:26.666 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-17T23:36:26.920 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-17T23:36:27.108 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-17T23:36:27.177 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-17T23:36:27.221 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-17T23:36:27.347 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-17T23:36:27.424 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-17T23:36:27.479 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-17T23:36:27.553 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-17T23:36:27.584 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-17T23:36:27.620 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-17T23:36:27.666 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-17T23:36:27.702 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-17T23:36:27.746 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-17T23:36:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-17T23:36:27.921 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-17T23:36:27.968 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-17T23:36:27.999 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-17T23:36:28.040 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-17T23:36:28.135 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-17T23:36:28.173 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-17T23:36:28.238 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-17T23:36:28.264 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-17T23:36:28.297 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-17T23:36:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-17T23:36:28.576 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-17T23:36:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-17T23:36:28.946 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-17T23:36:28.996 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-17T23:36:29.060 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-17T23:36:29.244 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-17T23:36:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-17T23:36:29.423 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-17T23:36:29.465 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-17T23:36:29.506 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-17T23:36:29.537 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-17T23:36:29.565 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-17T23:36:29.599 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-17T23:36:29.638 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-17T23:36:29.670 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-17T23:36:29.700 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-17T23:36:29.767 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-17T23:36:29.810 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-17T23:36:29.843 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-17T23:36:29.885 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-17T23:36:29.920 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-17T23:36:30.010 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-17T23:36:30.099 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-17T23:36:30.150 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-17T23:36:30.182 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-17T23:36:30.229 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-17T23:36:30.326 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-17T23:36:30.375 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-17T23:36:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-17T23:36:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-17T23:36:31.181 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-17T23:36:31.229 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-17T23:36:31.317 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-17T23:36:31.356 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-17T23:36:31.532 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-17T23:36:31.563 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-17T23:36:31.631 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-17T23:36:32.277 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-17T23:36:32.401 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-17T23:36:33.614 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-17T23:36:33.698 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-17T23:36:33.729 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-17T23:36:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-17T23:36:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-17T23:36:34.100 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-17T23:36:34.359 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-17T23:36:34.441 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-17T23:36:34.472 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-17T23:36:34.500 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-17T23:36:34.958 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-17T23:36:34.987 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-17T23:36:35.049 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-17T23:36:35.072 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-17T23:36:37.339 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:36:37.377 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:36:37.912 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:36:37.946 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-17T23:36:38.028 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-17T23:36:38.028 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-17T23:36:38.028 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-17T23:36:38.028 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:36:38.056 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-17T23:36:50.021 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-17T23:36:50.021 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2024-03-17T23:36:50.022 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2024-03-17T23:36:50.022 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2024-03-17T23:36:50.022 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2024-03-17T23:36:50.022 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2024-03-17T23:36:50.022 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2024-03-17T23:36:50.022 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2024-03-17T23:36:50.022 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:36:50.184 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-17T23:36:50.199 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-17T23:36:50.199 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:36:50.199 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-17T23:36:50.199 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:36:51.697 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-17T23:36:51.722 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-17T23:36:51.722 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:36:51.722 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-17T23:36:51.722 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-17T23:36:51.722 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-17T23:36:51.723 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:36:51.949 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-17T23:36:51.970 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-17T23:36:51.970 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:36:51.970 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-17T23:36:51.970 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-17T23:36:51.970 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-17T23:36:51.970 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:36:52.406 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-17T23:36:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-17T23:36:52.428 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:36:52.429 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-17T23:36:52.429 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-17T23:36:52.429 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-17T23:36:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:36:52.525 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-17T23:36:52.893 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-17T23:36:52.923 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-17T23:36:52.955 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-17T23:36:53.016 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-17T23:36:53.046 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-17T23:36:53.279 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-17T23:36:53.305 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-17T23:36:54.267 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-17T23:36:54.306 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-17T23:36:54.406 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-17T23:36:54.568 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-17T23:36:54.591 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-17T23:36:54.614 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-17T23:36:54.614 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:36:54.614 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-17T23:36:54.614 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-17T23:36:54.614 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-17T23:36:54.615 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:36:54.658 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-17T23:36:54.669 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-17T23:36:54.748 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-17T23:36:55.616 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-17T23:36:55.652 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-17T23:36:55.674 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-17T23:36:55.674 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:36:55.674 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-17T23:36:55.674 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-17T23:36:55.675 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-17T23:36:55.675 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:36:55.719 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-17T23:36:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-17T23:36:55.741 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:36:55.741 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-17T23:36:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:36:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:36:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:36:55.809 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:36:55.809 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-17T23:36:55.809 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-17T23:36:55.809 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-17T23:36:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:36:56.121 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-17T23:36:56.143 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-17T23:36:56.144 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:36:56.144 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-17T23:36:56.144 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-17T23:36:56.144 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-17T23:36:56.144 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:37:00.566 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-17T23:37:00.607 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-17T23:37:00.646 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-17T23:37:00.673 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-17T23:37:00.859 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-17T23:37:00.972 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-17T23:37:01.022 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-17T23:37:01.061 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-17T23:37:01.105 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-17T23:37:01.148 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-17T23:37:01.180 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-17T23:37:01.223 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-17T23:37:01.251 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-17T23:37:01.314 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-17T23:37:01.378 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-17T23:37:01.407 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-17T23:37:01.928 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:37:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:37:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:37:02.819 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-17T23:37:02.892 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-17T23:37:02.933 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-17T23:37:03.013 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-17T23:37:03.045 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-17T23:37:03.071 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-17T23:37:03.096 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-17T23:37:03.122 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-17T23:37:03.122 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-17T23:37:03.219 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-17T23:37:03.248 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-17T23:37:03.249 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-17T23:37:03.333 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-17T23:37:03.359 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-17T23:37:03.400 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-17T23:37:03.426 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-17T23:37:03.451 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-17T23:37:03.484 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-17T23:37:03.532 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-17T23:37:03.572 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-17T23:37:03.598 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-17T23:37:03.623 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-17T23:37:03.648 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-17T23:37:03.674 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-17T23:37:03.685 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:37:03.685 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:37:03.869 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:37:03.869 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-17T23:37:03.951 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-17T23:37:03.978 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-17T23:37:04.001 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-17T23:37:04.034 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-17T23:37:04.068 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-17T23:37:04.132 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-17T23:37:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-17T23:37:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-17T23:37:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-17T23:37:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-17T23:37:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-17T23:37:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-17T23:37:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-17T23:37:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-17T23:37:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-17T23:37:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-17T23:37:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:37:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:37:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-17T23:37:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-17T23:37:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-17T23:37:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-17T23:37:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:37:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-17T23:37:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-17T23:37:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-17T23:37:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-17T23:37:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-17T23:37:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-17T23:37:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-17T23:37:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-17T23:37:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-17T23:37:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-17T23:37:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-17T23:37:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-17T23:37:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-17T23:37:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-17T23:37:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-17T23:37:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-17T23:37:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-17T23:37:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-17T23:37:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-17T23:37:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-17T23:37:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-17T23:37:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-17T23:37:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-17T23:37:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-17T23:37:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-17T23:37:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-17T23:37:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-17T23:37:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-17T23:37:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-17T23:37:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-17T23:37:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-17T23:37:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-17T23:37:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-17T23:37:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-17T23:37:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-17T23:37:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-17T23:37:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-17T23:37:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-17T23:37:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-17T23:37:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-17T23:37:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-17T23:37:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-17T23:37:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-17T23:37:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-17T23:37:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-17T23:37:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-17T23:37:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-17T23:37:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-17T23:37:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-17T23:37:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-17T23:37:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-17T23:37:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-17T23:37:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-17T23:37:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-17T23:37:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-17T23:37:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-17T23:37:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-17T23:37:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-17T23:37:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-17T23:37:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-17T23:37:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-17T23:37:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-17T23:37:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-17T23:37:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-17T23:37:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-17T23:37:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-17T23:37:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-17T23:37:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-17T23:37:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-17T23:37:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-17T23:37:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-17T23:37:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-17T23:37:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-17T23:37:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-17T23:37:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-17T23:37:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-17T23:37:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-17T23:37:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-17T23:37:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-17T23:37:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-17T23:37:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-17T23:37:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-17T23:37:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-17T23:37:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-17T23:37:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-17T23:37:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-17T23:37:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-17T23:37:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-17T23:37:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-17T23:37:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-17T23:37:12.131 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-17T23:37:12.131 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:37:12.131 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2024-03-17T23:37:12.131 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-17T23:37:12.131 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-17T23:37:12.131 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-17T23:37:12.131 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-17T23:37:12.131 INFO:teuthology.orchestra.run.smithi191.stdout: acl-2.3.1-4.el9.x86_64 2024-03-17T23:37:12.131 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-17T23:37:12.132 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-17T23:37:12.132 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-17T23:37:12.132 INFO:teuthology.orchestra.run.smithi191.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-17T23:37:12.132 INFO:teuthology.orchestra.run.smithi191.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-17T23:37:12.132 INFO:teuthology.orchestra.run.smithi191.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-17T23:37:12.132 INFO:teuthology.orchestra.run.smithi191.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-17T23:37:12.132 INFO:teuthology.orchestra.run.smithi191.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-17T23:37:12.132 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.132 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.132 INFO:teuthology.orchestra.run.smithi191.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-17T23:37:12.132 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-17T23:37:12.133 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-17T23:37:12.133 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-17T23:37:12.133 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-17T23:37:12.133 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-17T23:37:12.133 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-17T23:37:12.133 INFO:teuthology.orchestra.run.smithi191.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-17T23:37:12.133 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-17T23:37:12.133 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-17T23:37:12.133 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-17T23:37:12.133 INFO:teuthology.orchestra.run.smithi191.stdout: automake-1.16.2-8.el9.noarch 2024-03-17T23:37:12.133 INFO:teuthology.orchestra.run.smithi191.stdout: bison-3.7.4-5.el9.x86_64 2024-03-17T23:37:12.133 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-17T23:37:12.133 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:37:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:37:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:37:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:37:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:37:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:37:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.135 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.135 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:37:12.135 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.135 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.135 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.135 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:37:12.135 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:37:12.135 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.135 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.135 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:37:12.135 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-17T23:37:12.135 INFO:teuthology.orchestra.run.smithi191.stdout: flex-2.6.4-9.el9.x86_64 2024-03-17T23:37:12.135 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: jq-1.6-16.el9.x86_64 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-17T23:37:12.136 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-17T23:37:12.137 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-17T23:37:12.137 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-17T23:37:12.137 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-17T23:37:12.137 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.137 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.137 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-17T23:37:12.137 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.137 INFO:teuthology.orchestra.run.smithi191.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-17T23:37:12.137 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-17T23:37:12.138 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-17T23:37:12.138 INFO:teuthology.orchestra.run.smithi191.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-17T23:37:12.138 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-17T23:37:12.138 INFO:teuthology.orchestra.run.smithi191.stdout: lua-5.4.4-4.el9.x86_64 2024-03-17T23:37:12.138 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-17T23:37:12.138 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-17T23:37:12.138 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-17T23:37:12.138 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-17T23:37:12.138 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-17T23:37:12.138 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-17T23:37:12.138 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-17T23:37:12.138 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-13.11-1.el9.x86_64 2024-03-17T23:37:12.139 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-17T23:37:12.139 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-17T23:37:12.139 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-17T23:37:12.139 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-17T23:37:12.139 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-17T23:37:12.139 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-17T23:37:12.139 INFO:teuthology.orchestra.run.smithi191.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-17T23:37:12.139 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-17T23:37:12.139 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-17T23:37:12.139 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-17T23:37:12.139 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.139 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.139 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-17T23:37:12.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-17T23:37:12.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-17T23:37:12.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-17T23:37:12.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-17T23:37:12.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-17T23:37:12.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-17T23:37:12.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-17T23:37:12.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-17T23:37:12.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-17T23:37:12.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-17T23:37:12.140 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-17T23:37:12.141 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-17T23:37:12.142 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-17T23:37:12.142 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-17T23:37:12.142 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-17T23:37:12.142 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-17T23:37:12.142 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2024-03-17T23:37:12.142 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-17T23:37:12.142 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-17T23:37:12.142 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-17T23:37:12.142 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-17T23:37:12.142 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-17T23:37:12.142 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-17T23:37:12.142 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-17T23:37:12.142 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-17T23:37:12.143 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-17T23:37:12.143 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.143 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.143 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-17T23:37:12.143 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-17T23:37:12.143 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-17T23:37:12.143 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.143 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-17T23:37:12.143 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-17T23:37:12.143 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-17T23:37:12.143 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-17T23:37:12.143 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-17T23:37:12.143 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-17T23:37:12.145 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-17T23:37:12.145 INFO:teuthology.orchestra.run.smithi191.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-17T23:37:12.145 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-17T23:37:12.145 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-17T23:37:12.145 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-17T23:37:12.145 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-17T23:37:12.145 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-17T23:37:12.145 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-17T23:37:12.146 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-17T23:37:12.146 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.146 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.146 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:37:12.146 INFO:teuthology.orchestra.run.smithi191.stdout: re2-1:20211101-3.el9.x86_64 2024-03-17T23:37:12.146 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-17T23:37:12.146 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-17T23:37:12.146 INFO:teuthology.orchestra.run.smithi191.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-17T23:37:12.146 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-17T23:37:12.146 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-17T23:37:12.147 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-17T23:37:12.147 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:37:12.147 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-17T23:37:12.619 DEBUG:teuthology.parallel:result is None 2024-03-17T23:40:34.671 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 21 kB/s | 6.1 MB 05:02 2024-03-17T23:40:34.989 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 21 kB/s | 6.1 MB 05:02 2024-03-17T23:40:36.606 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-03-17T23:40:36.730 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-03-17T23:40:38.419 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-17T23:40:38.690 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-17T23:40:43.548 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-17T23:40:44.346 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-17T23:40:45.975 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-17T23:40:45.975 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-17T23:40:45.987 INFO:teuthology.orchestra.run.smithi079.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-17T23:40:45.988 INFO:teuthology.orchestra.run.smithi079.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-17T23:40:45.989 INFO:teuthology.orchestra.run.smithi079.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-17T23:40:45.990 INFO:teuthology.orchestra.run.smithi079.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-17T23:40:45.991 INFO:teuthology.orchestra.run.smithi079.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-17T23:40:45.994 INFO:teuthology.orchestra.run.smithi079.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-17T23:40:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-17T23:40:46.074 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-17T23:40:46.096 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================ 2024-03-17T23:40:46.096 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-17T23:40:46.096 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================ 2024-03-17T23:40:46.097 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-17T23:40:46.097 INFO:teuthology.orchestra.run.smithi079.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-17T23:40:46.097 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-17T23:40:46.097 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-17T23:40:46.097 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-17T23:40:46.097 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-17T23:40:46.097 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-17T23:40:46.097 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-17T23:40:46.097 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-17T23:40:46.097 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-17T23:40:46.097 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-17T23:40:46.097 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-17T23:40:46.097 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-17T23:40:46.098 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-17T23:40:46.098 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-17T23:40:46.098 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-17T23:40:46.098 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-17T23:40:46.098 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-17T23:40:46.098 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-17T23:40:46.098 INFO:teuthology.orchestra.run.smithi079.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-17T23:40:46.098 INFO:teuthology.orchestra.run.smithi079.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-17T23:40:46.098 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-17T23:40:46.098 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-17T23:40:46.098 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-17T23:40:46.098 INFO:teuthology.orchestra.run.smithi079.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-17T23:40:46.099 INFO:teuthology.orchestra.run.smithi079.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-17T23:40:46.099 INFO:teuthology.orchestra.run.smithi079.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-17T23:40:46.099 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-17T23:40:46.099 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-17T23:40:46.099 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-17T23:40:46.099 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-17T23:40:46.099 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-17T23:40:46.099 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-17T23:40:46.099 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-17T23:40:46.099 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-17T23:40:46.099 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-17T23:40:46.100 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-17T23:40:46.100 INFO:teuthology.orchestra.run.smithi079.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-17T23:40:46.100 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2024-03-17T23:40:46.100 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-17T23:40:46.100 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-17T23:40:46.100 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-17T23:40:46.100 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-17T23:40:46.100 INFO:teuthology.orchestra.run.smithi079.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-17T23:40:46.100 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-17T23:40:46.100 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-17T23:40:46.100 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-17T23:40:46.100 INFO:teuthology.orchestra.run.smithi079.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-17T23:40:46.101 INFO:teuthology.orchestra.run.smithi079.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-17T23:40:46.101 INFO:teuthology.orchestra.run.smithi079.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-17T23:40:46.101 INFO:teuthology.orchestra.run.smithi079.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-17T23:40:46.101 INFO:teuthology.orchestra.run.smithi079.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-17T23:40:46.101 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-17T23:40:46.101 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-17T23:40:46.101 INFO:teuthology.orchestra.run.smithi079.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-17T23:40:46.101 INFO:teuthology.orchestra.run.smithi079.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-17T23:40:46.101 INFO:teuthology.orchestra.run.smithi079.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-17T23:40:46.101 INFO:teuthology.orchestra.run.smithi079.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-17T23:40:46.101 INFO:teuthology.orchestra.run.smithi079.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-17T23:40:46.101 INFO:teuthology.orchestra.run.smithi079.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-17T23:40:46.101 INFO:teuthology.orchestra.run.smithi079.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-17T23:40:46.102 INFO:teuthology.orchestra.run.smithi079.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-17T23:40:46.102 INFO:teuthology.orchestra.run.smithi079.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-17T23:40:46.102 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-17T23:40:46.102 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-17T23:40:46.102 INFO:teuthology.orchestra.run.smithi079.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-17T23:40:46.102 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-17T23:40:46.102 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-17T23:40:46.102 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-17T23:40:46.102 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-17T23:40:46.102 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-17T23:40:46.102 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-17T23:40:46.102 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-17T23:40:46.103 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-17T23:40:46.103 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-17T23:40:46.103 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-17T23:40:46.103 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-17T23:40:46.103 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-17T23:40:46.103 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-17T23:40:46.103 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-17T23:40:46.103 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-17T23:40:46.103 INFO:teuthology.orchestra.run.smithi079.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-17T23:40:46.103 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-17T23:40:46.103 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-17T23:40:46.104 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-17T23:40:46.104 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-17T23:40:46.104 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-17T23:40:46.104 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-17T23:40:46.104 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-17T23:40:46.104 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-17T23:40:46.104 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-17T23:40:46.104 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-17T23:40:46.104 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-17T23:40:46.104 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-17T23:40:46.104 INFO:teuthology.orchestra.run.smithi079.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-17T23:40:46.105 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-17T23:40:46.105 INFO:teuthology.orchestra.run.smithi079.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-17T23:40:46.105 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-17T23:40:46.105 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-17T23:40:46.105 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-17T23:40:46.105 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-17T23:40:46.105 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-17T23:40:46.105 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-17T23:40:46.105 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-17T23:40:46.105 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-17T23:40:46.105 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-17T23:40:46.105 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-17T23:40:46.105 INFO:teuthology.orchestra.run.smithi079.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-17T23:40:46.106 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-17T23:40:46.106 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-17T23:40:46.106 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-17T23:40:46.106 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-17T23:40:46.106 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-17T23:40:46.106 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-17T23:40:46.106 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-17T23:40:46.106 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-17T23:40:46.106 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-17T23:40:46.106 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-17T23:40:46.106 INFO:teuthology.orchestra.run.smithi079.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-17T23:40:46.106 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-17T23:40:46.106 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-17T23:40:46.107 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-17T23:40:46.107 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-17T23:40:46.107 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-17T23:40:46.107 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-17T23:40:46.107 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-17T23:40:46.107 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-17T23:40:46.107 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-17T23:40:46.107 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-17T23:40:46.107 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-17T23:40:46.107 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-17T23:40:46.107 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-17T23:40:46.107 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-17T23:40:46.107 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-17T23:40:46.108 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-17T23:40:46.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-17T23:40:46.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-17T23:40:46.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-17T23:40:46.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-17T23:40:46.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-17T23:40:46.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-17T23:40:46.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-17T23:40:46.110 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-17T23:40:46.110 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-17T23:40:46.110 INFO:teuthology.orchestra.run.smithi079.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-17T23:40:46.110 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-17T23:40:46.110 INFO:teuthology.orchestra.run.smithi079.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-17T23:40:46.110 INFO:teuthology.orchestra.run.smithi079.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-17T23:40:46.110 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-17T23:40:46.111 INFO:teuthology.orchestra.run.smithi079.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-17T23:40:46.111 INFO:teuthology.orchestra.run.smithi079.stdout:Installing weak dependencies: 2024-03-17T23:40:46.111 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-17T23:40:46.111 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:40:46.111 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-17T23:40:46.111 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================ 2024-03-17T23:40:46.111 INFO:teuthology.orchestra.run.smithi079.stdout:Install 150 Packages 2024-03-17T23:40:46.111 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 24 Packages 2024-03-17T23:40:46.111 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:40:46.112 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 224 M 2024-03-17T23:40:46.112 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-17T23:40:46.949 INFO:teuthology.orchestra.run.smithi079.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-03-17T23:40:47.140 INFO:teuthology.orchestra.run.smithi012.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-17T23:40:47.142 INFO:teuthology.orchestra.run.smithi012.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-17T23:40:47.162 INFO:teuthology.orchestra.run.smithi012.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-17T23:40:47.164 INFO:teuthology.orchestra.run.smithi012.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-17T23:40:47.165 INFO:teuthology.orchestra.run.smithi012.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-17T23:40:47.166 INFO:teuthology.orchestra.run.smithi012.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-17T23:40:47.169 INFO:teuthology.orchestra.run.smithi012.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-17T23:40:47.173 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-17T23:40:47.174 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-17T23:40:47.270 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-17T23:40:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-17T23:40:47.301 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-17T23:40:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-17T23:40:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-17T23:40:47.301 INFO:teuthology.orchestra.run.smithi012.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-17T23:40:47.301 INFO:teuthology.orchestra.run.smithi012.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-17T23:40:47.302 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-17T23:40:47.302 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-17T23:40:47.302 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-17T23:40:47.302 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-17T23:40:47.302 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-17T23:40:47.302 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-17T23:40:47.302 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-17T23:40:47.302 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-17T23:40:47.302 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-17T23:40:47.302 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-17T23:40:47.302 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-17T23:40:47.302 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-17T23:40:47.303 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-17T23:40:47.304 INFO:teuthology.orchestra.run.smithi012.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-17T23:40:47.305 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-17T23:40:47.306 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-17T23:40:47.306 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-17T23:40:47.306 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-17T23:40:47.306 INFO:teuthology.orchestra.run.smithi012.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-17T23:40:47.306 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-17T23:40:47.306 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-17T23:40:47.306 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-17T23:40:47.306 INFO:teuthology.orchestra.run.smithi012.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-17T23:40:47.306 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-17T23:40:47.306 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-17T23:40:47.306 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-17T23:40:47.306 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-17T23:40:47.306 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-17T23:40:47.307 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-17T23:40:47.308 INFO:teuthology.orchestra.run.smithi012.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-17T23:40:47.309 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-17T23:40:47.310 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-17T23:40:47.311 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-17T23:40:47.312 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-17T23:40:47.313 INFO:teuthology.orchestra.run.smithi012.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-17T23:40:47.314 INFO:teuthology.orchestra.run.smithi012.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-17T23:40:47.314 INFO:teuthology.orchestra.run.smithi012.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-17T23:40:47.314 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-17T23:40:47.314 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-17T23:40:47.314 INFO:teuthology.orchestra.run.smithi012.stdout:Installing weak dependencies: 2024-03-17T23:40:47.314 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-17T23:40:47.314 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:40:47.314 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-17T23:40:47.314 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-17T23:40:47.314 INFO:teuthology.orchestra.run.smithi012.stdout:Install 150 Packages 2024-03-17T23:40:47.314 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 24 Packages 2024-03-17T23:40:47.314 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:40:47.315 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 224 M 2024-03-17T23:40:47.315 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-17T23:40:47.991 INFO:teuthology.orchestra.run.smithi079.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 855 kB/s | 890 kB 00:01 2024-03-17T23:40:48.191 INFO:teuthology.orchestra.run.smithi079.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 724 kB/s | 145 kB 00:00 2024-03-17T23:40:49.071 INFO:teuthology.orchestra.run.smithi012.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 5.9 kB/s | 6.6 kB 00:01 2024-03-17T23:40:50.191 INFO:teuthology.orchestra.run.smithi079.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 1.2 MB/s | 2.4 MB 00:01 2024-03-17T23:40:51.549 INFO:teuthology.orchestra.run.smithi079.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 1.1 MB/s | 1.5 MB 00:01 2024-03-17T23:40:51.616 INFO:teuthology.orchestra.run.smithi079.stdout:(6/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 1.1 MB/s | 5.4 MB 00:05 2024-03-17T23:40:52.372 INFO:teuthology.orchestra.run.smithi012.stdout:(2/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 1.2 MB/s | 5.4 MB 00:04 2024-03-17T23:40:52.630 INFO:teuthology.orchestra.run.smithi012.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 562 kB/s | 145 kB 00:00 2024-03-17T23:40:53.405 INFO:teuthology.orchestra.run.smithi012.stdout:(4/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 205 kB/s | 890 kB 00:04 2024-03-17T23:40:55.114 INFO:teuthology.orchestra.run.smithi012.stdout:(5/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 974 kB/s | 2.4 MB 00:02 2024-03-17T23:40:59.005 INFO:teuthology.orchestra.run.smithi012.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 273 kB/s | 1.5 MB 00:05 2024-03-17T23:41:00.105 INFO:teuthology.orchestra.run.smithi012.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 964 kB/s | 4.7 MB 00:04 2024-03-17T23:41:04.097 INFO:teuthology.orchestra.run.smithi079.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 383 kB/s | 4.7 MB 00:12 2024-03-17T23:41:07.822 INFO:teuthology.orchestra.run.smithi079.stdout:(8/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 995 kB/s | 21 MB 00:21 2024-03-17T23:41:07.989 INFO:teuthology.orchestra.run.smithi079.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 151 kB/s | 25 kB 00:00 2024-03-17T23:41:12.819 INFO:teuthology.orchestra.run.smithi012.stdout:(8/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 792 kB/s | 9.8 MB 00:12 2024-03-17T23:41:12.928 INFO:teuthology.orchestra.run.smithi012.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 235 kB/s | 25 kB 00:00 2024-03-17T23:41:13.380 INFO:teuthology.orchestra.run.smithi079.stdout:(10/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 792 kB/s | 17 MB 00:21 2024-03-17T23:41:13.963 INFO:teuthology.orchestra.run.smithi079.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 380 kB/s | 221 kB 00:00 2024-03-17T23:41:14.063 INFO:teuthology.orchestra.run.smithi079.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 271 kB/s | 27 kB 00:00 2024-03-17T23:41:14.163 INFO:teuthology.orchestra.run.smithi079.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 317 kB/s | 32 kB 00:00 2024-03-17T23:41:15.405 INFO:teuthology.orchestra.run.smithi079.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 601 kB/s | 747 kB 00:01 2024-03-17T23:41:15.656 INFO:teuthology.orchestra.run.smithi079.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 654 kB/s | 163 kB 00:00 2024-03-17T23:41:15.881 INFO:teuthology.orchestra.run.smithi079.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 564 kB/s | 127 kB 00:00 2024-03-17T23:41:16.523 INFO:teuthology.orchestra.run.smithi079.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 781 kB/s | 501 kB 00:00 2024-03-17T23:41:16.744 INFO:teuthology.orchestra.run.smithi012.stdout:(10/174): ceph-common-19.0.0-2162.g3d78f5ba.el9 734 kB/s | 21 MB 00:28 2024-03-17T23:41:17.153 INFO:teuthology.orchestra.run.smithi012.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 544 kB/s | 221 kB 00:00 2024-03-17T23:41:17.245 INFO:teuthology.orchestra.run.smithi012.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 298 kB/s | 27 kB 00:00 2024-03-17T23:41:17.336 INFO:teuthology.orchestra.run.smithi012.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 349 kB/s | 32 kB 00:00 2024-03-17T23:41:18.287 INFO:teuthology.orchestra.run.smithi012.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 787 kB/s | 747 kB 00:00 2024-03-17T23:41:18.537 INFO:teuthology.orchestra.run.smithi012.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 656 kB/s | 163 kB 00:00 2024-03-17T23:41:18.720 INFO:teuthology.orchestra.run.smithi012.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 695 kB/s | 127 kB 00:00 2024-03-17T23:41:19.312 INFO:teuthology.orchestra.run.smithi012.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 848 kB/s | 501 kB 00:00 2024-03-17T23:41:21.980 INFO:teuthology.orchestra.run.smithi079.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 964 kB/s | 5.1 MB 00:05 2024-03-17T23:41:22.106 INFO:teuthology.orchestra.run.smithi079.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 362 kB/s | 45 kB 00:00 2024-03-17T23:41:22.289 INFO:teuthology.orchestra.run.smithi079.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 715 kB/s | 131 kB 00:00 2024-03-17T23:41:22.465 INFO:teuthology.orchestra.run.smithi079.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 927 kB/s | 162 kB 00:00 2024-03-17T23:41:22.781 INFO:teuthology.orchestra.run.smithi079.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 1.0 MB/s | 320 kB 00:00 2024-03-17T23:41:23.090 INFO:teuthology.orchestra.run.smithi079.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 966 kB/s | 298 kB 00:00 2024-03-17T23:41:23.240 INFO:teuthology.orchestra.run.smithi079.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 664 kB/s | 99 kB 00:00 2024-03-17T23:41:23.332 INFO:teuthology.orchestra.run.smithi079.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 930 kB/s | 85 kB 00:00 2024-03-17T23:41:23.529 INFO:teuthology.orchestra.run.smithi012.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 1.2 MB/s | 5.1 MB 00:04 2024-03-17T23:41:23.603 INFO:teuthology.orchestra.run.smithi012.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 611 kB/s | 45 kB 00:00 2024-03-17T23:41:23.746 INFO:teuthology.orchestra.run.smithi012.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 932 kB/s | 131 kB 00:00 2024-03-17T23:41:23.896 INFO:teuthology.orchestra.run.smithi012.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.1 MB/s | 162 kB 00:00 2024-03-17T23:41:24.121 INFO:teuthology.orchestra.run.smithi012.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 1.4 MB/s | 320 kB 00:00 2024-03-17T23:41:24.355 INFO:teuthology.orchestra.run.smithi012.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 1.3 MB/s | 298 kB 00:00 2024-03-17T23:41:24.455 INFO:teuthology.orchestra.run.smithi012.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.0 MB/s | 99 kB 00:00 2024-03-17T23:41:24.547 INFO:teuthology.orchestra.run.smithi012.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 941 kB/s | 85 kB 00:00 2024-03-17T23:41:25.440 INFO:teuthology.orchestra.run.smithi079.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 1.5 MB/s | 3.1 MB 00:02 2024-03-17T23:41:25.549 INFO:teuthology.orchestra.run.smithi079.stdout:(27/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 469 kB/s | 9.8 MB 00:21 2024-03-17T23:41:25.641 INFO:teuthology.orchestra.run.smithi079.stdout:(28/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 852 kB/s | 170 kB 00:00 2024-03-17T23:41:25.666 INFO:teuthology.orchestra.run.smithi079.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2162.g 291 kB/s | 34 kB 00:00 2024-03-17T23:41:25.816 INFO:teuthology.orchestra.run.smithi079.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 792 kB/s | 139 kB 00:00 2024-03-17T23:41:26.138 INFO:teuthology.orchestra.run.smithi012.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 1.9 MB/s | 3.1 MB 00:01 2024-03-17T23:41:26.256 INFO:teuthology.orchestra.run.smithi012.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.4 MB/s | 170 kB 00:00 2024-03-17T23:41:26.356 INFO:teuthology.orchestra.run.smithi012.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 342 kB/s | 34 kB 00:00 2024-03-17T23:41:26.464 INFO:teuthology.orchestra.run.smithi012.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.3 MB/s | 139 kB 00:00 2024-03-17T23:41:27.165 INFO:teuthology.orchestra.run.smithi012.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 2.6 MB/s | 1.8 MB 00:00 2024-03-17T23:41:28.999 INFO:teuthology.orchestra.run.smithi079.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 551 kB/s | 1.8 MB 00:03 2024-03-17T23:41:29.550 INFO:teuthology.orchestra.run.smithi079.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 452 kB/s | 248 kB 00:00 2024-03-17T23:41:29.658 INFO:teuthology.orchestra.run.smithi079.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 455 kB/s | 49 kB 00:00 2024-03-17T23:41:29.690 INFO:teuthology.orchestra.run.smithi012.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 2.9 MB/s | 7.4 MB 00:02 2024-03-17T23:41:29.758 INFO:teuthology.orchestra.run.smithi079.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 164 kB/s | 16 kB 00:00 2024-03-17T23:41:30.065 INFO:teuthology.orchestra.run.smithi012.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 665 kB/s | 248 kB 00:00 2024-03-17T23:41:30.165 INFO:teuthology.orchestra.run.smithi012.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 497 kB/s | 49 kB 00:00 2024-03-17T23:41:30.207 INFO:teuthology.orchestra.run.smithi079.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 588 kB/s | 264 kB 00:00 2024-03-17T23:41:30.265 INFO:teuthology.orchestra.run.smithi012.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 166 kB/s | 16 kB 00:00 2024-03-17T23:41:30.657 INFO:teuthology.orchestra.run.smithi012.stdout:(35/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 545 kB/s | 17 MB 00:31 2024-03-17T23:41:30.682 INFO:teuthology.orchestra.run.smithi012.stdout:(36/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 634 kB/s | 264 kB 00:00 2024-03-17T23:41:30.782 INFO:teuthology.orchestra.run.smithi012.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 183 kB/s | 18 kB 00:00 2024-03-17T23:41:30.983 INFO:teuthology.orchestra.run.smithi012.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 662 kB 00:00 2024-03-17T23:41:31.050 INFO:teuthology.orchestra.run.smithi012.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2024-03-17T23:41:31.083 INFO:teuthology.orchestra.run.smithi012.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.2 MB/s | 106 kB 00:00 2024-03-17T23:41:31.125 INFO:teuthology.orchestra.run.smithi012.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 307 kB 00:00 2024-03-17T23:41:31.151 INFO:teuthology.orchestra.run.smithi012.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 1.3 MB/s | 32 kB 00:00 2024-03-17T23:41:31.268 INFO:teuthology.orchestra.run.smithi012.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-17T23:41:31.293 INFO:teuthology.orchestra.run.smithi012.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 716 kB/s | 17 kB 00:00 2024-03-17T23:41:31.307 INFO:teuthology.orchestra.run.smithi079.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 681 kB/s | 748 kB 00:01 2024-03-17T23:41:31.318 INFO:teuthology.orchestra.run.smithi012.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 411 kB/s | 10 kB 00:00 2024-03-17T23:41:31.360 INFO:teuthology.orchestra.run.smithi012.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-03-17T23:41:31.394 INFO:teuthology.orchestra.run.smithi012.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-17T23:41:31.416 INFO:teuthology.orchestra.run.smithi079.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 168 kB/s | 18 kB 00:00 2024-03-17T23:41:31.461 INFO:teuthology.orchestra.run.smithi012.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-03-17T23:41:31.519 INFO:teuthology.orchestra.run.smithi012.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-17T23:41:31.549 INFO:teuthology.orchestra.run.smithi079.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.9 MB/s | 662 kB 00:00 2024-03-17T23:41:31.553 INFO:teuthology.orchestra.run.smithi012.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-17T23:41:31.587 INFO:teuthology.orchestra.run.smithi012.stdout:(51/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 806 kB/s | 748 kB 00:00 2024-03-17T23:41:31.600 INFO:teuthology.orchestra.run.smithi079.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-17T23:41:31.620 INFO:teuthology.orchestra.run.smithi012.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-17T23:41:31.634 INFO:teuthology.orchestra.run.smithi079.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-17T23:41:31.654 INFO:teuthology.orchestra.run.smithi012.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-03-17T23:41:31.675 INFO:teuthology.orchestra.run.smithi079.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.3 MB/s | 307 kB 00:00 2024-03-17T23:41:31.696 INFO:teuthology.orchestra.run.smithi012.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.9 MB/s | 217 kB 00:00 2024-03-17T23:41:31.709 INFO:teuthology.orchestra.run.smithi079.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 945 kB/s | 32 kB 00:00 2024-03-17T23:41:31.722 INFO:teuthology.orchestra.run.smithi012.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 528 kB/s | 34 kB 00:00 2024-03-17T23:41:31.859 INFO:teuthology.orchestra.run.smithi079.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2024-03-17T23:41:31.880 INFO:teuthology.orchestra.run.smithi012.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-03-17T23:41:31.885 INFO:teuthology.orchestra.run.smithi079.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 690 kB/s | 17 kB 00:00 2024-03-17T23:41:31.918 INFO:teuthology.orchestra.run.smithi079.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 301 kB/s | 10 kB 00:00 2024-03-17T23:41:31.939 INFO:teuthology.orchestra.run.smithi012.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 7.7 MB/s | 1.5 MB 00:00 2024-03-17T23:41:31.952 INFO:teuthology.orchestra.run.smithi079.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-17T23:41:31.985 INFO:teuthology.orchestra.run.smithi079.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-17T23:41:31.989 INFO:teuthology.orchestra.run.smithi012.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 8.2 MB/s | 811 kB 00:00 2024-03-17T23:41:32.014 INFO:teuthology.orchestra.run.smithi012.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 1.9 MB/s | 134 kB 00:00 2024-03-17T23:41:32.028 INFO:teuthology.orchestra.run.smithi079.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 15 MB/s | 661 kB 00:00 2024-03-17T23:41:32.094 INFO:teuthology.orchestra.run.smithi079.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-17T23:41:32.128 INFO:teuthology.orchestra.run.smithi079.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-17T23:41:32.162 INFO:teuthology.orchestra.run.smithi079.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2024-03-17T23:41:32.181 INFO:teuthology.orchestra.run.smithi012.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-17T23:41:32.204 INFO:teuthology.orchestra.run.smithi079.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-17T23:41:32.223 INFO:teuthology.orchestra.run.smithi012.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.6 MB/s | 1.0 MB 00:00 2024-03-17T23:41:32.237 INFO:teuthology.orchestra.run.smithi079.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.4 MB/s | 217 kB 00:00 2024-03-17T23:41:32.288 INFO:teuthology.orchestra.run.smithi079.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 683 kB/s | 34 kB 00:00 2024-03-17T23:41:32.332 INFO:teuthology.orchestra.run.smithi012.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 725 kB/s | 78 kB 00:00 2024-03-17T23:41:32.424 INFO:teuthology.orchestra.run.smithi012.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-17T23:41:32.455 INFO:teuthology.orchestra.run.smithi079.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-03-17T23:41:32.466 INFO:teuthology.orchestra.run.smithi012.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 189 kB/s | 24 kB 00:00 2024-03-17T23:41:32.491 INFO:teuthology.orchestra.run.smithi012.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 275 kB/s | 18 kB 00:00 2024-03-17T23:41:32.516 INFO:teuthology.orchestra.run.smithi012.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-03-17T23:41:32.522 INFO:teuthology.orchestra.run.smithi079.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 23 MB/s | 1.5 MB 00:00 2024-03-17T23:41:32.541 INFO:teuthology.orchestra.run.smithi012.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 656 kB/s | 32 kB 00:00 2024-03-17T23:41:32.588 INFO:teuthology.orchestra.run.smithi079.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-17T23:41:32.631 INFO:teuthology.orchestra.run.smithi079.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 3.1 MB/s | 134 kB 00:00 2024-03-17T23:41:32.683 INFO:teuthology.orchestra.run.smithi012.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-17T23:41:32.708 INFO:teuthology.orchestra.run.smithi012.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-17T23:41:32.750 INFO:teuthology.orchestra.run.smithi012.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2024-03-17T23:41:32.776 INFO:teuthology.orchestra.run.smithi012.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-17T23:41:32.789 INFO:teuthology.orchestra.run.smithi079.stdout:(59/174): ceph-mgr-diskprediction-local-19.0.0- 1.1 MB/s | 7.4 MB 00:06 2024-03-17T23:41:32.809 INFO:teuthology.orchestra.run.smithi012.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-17T23:41:32.835 INFO:teuthology.orchestra.run.smithi012.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.1 MB/s | 238 kB 00:00 2024-03-17T23:41:32.856 INFO:teuthology.orchestra.run.smithi079.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 25 MB/s | 5.7 MB 00:00 2024-03-17T23:41:32.876 INFO:teuthology.orchestra.run.smithi012.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.2 MB/s | 132 kB 00:00 2024-03-17T23:41:32.890 INFO:teuthology.orchestra.run.smithi079.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-03-17T23:41:32.901 INFO:teuthology.orchestra.run.smithi012.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-17T23:41:32.915 INFO:teuthology.orchestra.run.smithi079.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-17T23:41:32.935 INFO:teuthology.orchestra.run.smithi012.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-17T23:41:32.952 INFO:teuthology.orchestra.run.smithi012.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 883 kB/s | 43 kB 00:00 2024-03-17T23:41:33.040 INFO:teuthology.orchestra.run.smithi079.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 39 MB/s | 5.8 MB 00:00 2024-03-17T23:41:33.044 INFO:teuthology.orchestra.run.smithi012.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 427 kB/s | 39 kB 00:00 2024-03-17T23:41:33.074 INFO:teuthology.orchestra.run.smithi079.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 156 kB/s | 24 kB 00:00 2024-03-17T23:41:33.099 INFO:teuthology.orchestra.run.smithi079.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 308 kB/s | 18 kB 00:00 2024-03-17T23:41:33.125 INFO:teuthology.orchestra.run.smithi079.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-17T23:41:33.150 INFO:teuthology.orchestra.run.smithi079.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 630 kB/s | 32 kB 00:00 2024-03-17T23:41:33.275 INFO:teuthology.orchestra.run.smithi079.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-17T23:41:33.300 INFO:teuthology.orchestra.run.smithi079.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-17T23:41:33.326 INFO:teuthology.orchestra.run.smithi079.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-17T23:41:33.343 INFO:teuthology.orchestra.run.smithi079.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 69 kB 00:00 2024-03-17T23:41:33.360 INFO:teuthology.orchestra.run.smithi079.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-17T23:41:33.369 INFO:teuthology.orchestra.run.smithi012.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-17T23:41:33.385 INFO:teuthology.orchestra.run.smithi079.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 5.5 MB/s | 238 kB 00:00 2024-03-17T23:41:33.386 INFO:teuthology.orchestra.run.smithi012.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 126 kB/s | 41 kB 00:00 2024-03-17T23:41:33.410 INFO:teuthology.orchestra.run.smithi079.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-17T23:41:33.444 INFO:teuthology.orchestra.run.smithi079.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-17T23:41:33.444 INFO:teuthology.orchestra.run.smithi012.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-17T23:41:33.461 INFO:teuthology.orchestra.run.smithi079.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 944 kB/s | 47 kB 00:00 2024-03-17T23:41:33.461 INFO:teuthology.orchestra.run.smithi012.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 765 kB/s | 56 kB 00:00 2024-03-17T23:41:33.478 INFO:teuthology.orchestra.run.smithi079.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-17T23:41:33.495 INFO:teuthology.orchestra.run.smithi012.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-17T23:41:33.520 INFO:teuthology.orchestra.run.smithi012.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 770 kB/s | 44 kB 00:00 2024-03-17T23:41:33.546 INFO:teuthology.orchestra.run.smithi012.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-17T23:41:33.596 INFO:teuthology.orchestra.run.smithi012.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.2 MB/s | 241 kB 00:00 2024-03-17T23:41:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 245 kB/s | 39 kB 00:00 2024-03-17T23:41:33.655 INFO:teuthology.orchestra.run.smithi012.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-17T23:41:33.680 INFO:teuthology.orchestra.run.smithi012.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-17T23:41:33.697 INFO:teuthology.orchestra.run.smithi012.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 3.0 MB/s | 124 kB 00:00 2024-03-17T23:41:33.731 INFO:teuthology.orchestra.run.smithi012.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.8 MB/s | 187 kB 00:00 2024-03-17T23:41:33.845 INFO:teuthology.orchestra.run.smithi079.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-17T23:41:33.870 INFO:teuthology.orchestra.run.smithi079.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 177 kB/s | 41 kB 00:00 2024-03-17T23:41:33.896 INFO:teuthology.orchestra.run.smithi079.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-17T23:41:33.931 INFO:teuthology.orchestra.run.smithi012.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 810 kB/s | 187 kB 00:00 2024-03-17T23:41:33.951 INFO:teuthology.orchestra.run.smithi079.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 707 kB/s | 56 kB 00:00 2024-03-17T23:41:33.979 INFO:teuthology.orchestra.run.smithi079.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 750 kB/s | 63 kB 00:00 2024-03-17T23:41:34.005 INFO:teuthology.orchestra.run.smithi079.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 821 kB/s | 44 kB 00:00 2024-03-17T23:41:34.015 INFO:teuthology.orchestra.run.smithi012.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-17T23:41:34.032 INFO:teuthology.orchestra.run.smithi079.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-17T23:41:34.065 INFO:teuthology.orchestra.run.smithi012.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-03-17T23:41:34.065 INFO:teuthology.orchestra.run.smithi079.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.9 MB/s | 241 kB 00:00 2024-03-17T23:41:34.098 INFO:teuthology.orchestra.run.smithi012.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-17T23:41:34.099 INFO:teuthology.orchestra.run.smithi079.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-17T23:41:34.125 INFO:teuthology.orchestra.run.smithi079.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-17T23:41:34.150 INFO:teuthology.orchestra.run.smithi079.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-17T23:41:34.175 INFO:teuthology.orchestra.run.smithi079.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 187 kB 00:00 2024-03-17T23:41:34.992 INFO:teuthology.orchestra.run.smithi079.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 222 kB/s | 187 kB 00:00 2024-03-17T23:41:35.817 INFO:teuthology.orchestra.run.smithi079.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 237 kB/s | 195 kB 00:00 2024-03-17T23:41:35.859 INFO:teuthology.orchestra.run.smithi079.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 477 kB/s | 803 kB 00:01 2024-03-17T23:41:36.026 INFO:teuthology.orchestra.run.smithi079.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 129 kB/s | 21 kB 00:00 2024-03-17T23:41:36.110 INFO:teuthology.orchestra.run.smithi079.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 305 kB/s | 25 kB 00:00 2024-03-17T23:41:36.328 INFO:teuthology.orchestra.run.smithi079.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 19 MB/s | 4.1 MB 00:00 2024-03-17T23:41:36.402 INFO:teuthology.orchestra.run.smithi079.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-03-17T23:41:36.469 INFO:teuthology.orchestra.run.smithi079.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-03-17T23:41:36.536 INFO:teuthology.orchestra.run.smithi079.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.2 MB/s | 216 kB 00:00 2024-03-17T23:41:36.595 INFO:teuthology.orchestra.run.smithi079.stdout:(100/174): python3-requests-2.25.1-8.el9.noarch 161 kB/s | 125 kB 00:00 2024-03-17T23:41:36.612 INFO:teuthology.orchestra.run.smithi079.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 463 kB/s | 35 kB 00:00 2024-03-17T23:41:36.646 INFO:teuthology.orchestra.run.smithi079.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 888 kB/s | 44 kB 00:00 2024-03-17T23:41:36.696 INFO:teuthology.orchestra.run.smithi079.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.1 MB/s | 94 kB 00:00 2024-03-17T23:41:36.904 INFO:teuthology.orchestra.run.smithi079.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-03-17T23:41:37.272 INFO:teuthology.orchestra.run.smithi079.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-03-17T23:41:37.397 INFO:teuthology.orchestra.run.smithi079.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.2 MB/s | 863 kB 00:00 2024-03-17T23:41:37.464 INFO:teuthology.orchestra.run.smithi079.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.6 MB/s | 309 kB 00:00 2024-03-17T23:41:37.656 INFO:teuthology.orchestra.run.smithi079.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-17T23:41:37.739 INFO:teuthology.orchestra.run.smithi079.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2024-03-17T23:41:37.798 INFO:teuthology.orchestra.run.smithi079.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-03-17T23:41:37.849 INFO:teuthology.orchestra.run.smithi079.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-17T23:41:37.991 INFO:teuthology.orchestra.run.smithi079.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 8.4 MB/s | 4.4 MB 00:00 2024-03-17T23:41:38.016 INFO:teuthology.orchestra.run.smithi079.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 905 kB/s | 151 kB 00:00 2024-03-17T23:41:38.133 INFO:teuthology.orchestra.run.smithi079.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-17T23:41:38.208 INFO:teuthology.orchestra.run.smithi079.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 580 kB/s | 43 kB 00:00 2024-03-17T23:41:38.242 INFO:teuthology.orchestra.run.smithi079.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 2.4 MB/s | 548 kB 00:00 2024-03-17T23:41:38.325 INFO:teuthology.orchestra.run.smithi079.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-03-17T23:41:38.351 INFO:teuthology.orchestra.run.smithi079.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-17T23:41:38.384 INFO:teuthology.orchestra.run.smithi079.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-17T23:41:38.485 INFO:teuthology.orchestra.run.smithi079.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-17T23:41:38.552 INFO:teuthology.orchestra.run.smithi079.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.8 MB/s | 303 kB 00:00 2024-03-17T23:41:38.577 INFO:teuthology.orchestra.run.smithi079.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 382 kB/s | 35 kB 00:00 2024-03-17T23:41:38.644 INFO:teuthology.orchestra.run.smithi079.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 2.2 MB/s | 203 kB 00:00 2024-03-17T23:41:38.836 INFO:teuthology.orchestra.run.smithi079.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 753 kB/s | 144 kB 00:00 2024-03-17T23:41:38.987 INFO:teuthology.orchestra.run.smithi079.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 71 kB/s | 11 kB 00:00 2024-03-17T23:41:39.020 INFO:teuthology.orchestra.run.smithi079.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.6 MB/s | 2.0 MB 00:00 2024-03-17T23:41:39.062 INFO:teuthology.orchestra.run.smithi079.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-03-17T23:41:39.121 INFO:teuthology.orchestra.run.smithi079.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-03-17T23:41:39.205 INFO:teuthology.orchestra.run.smithi079.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 236 kB/s | 20 kB 00:00 2024-03-17T23:41:39.230 INFO:teuthology.orchestra.run.smithi079.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 116 kB/s | 19 kB 00:00 2024-03-17T23:41:39.305 INFO:teuthology.orchestra.run.smithi079.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 618 kB/s | 46 kB 00:00 2024-03-17T23:41:39.364 INFO:teuthology.orchestra.run.smithi079.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-17T23:41:39.431 INFO:teuthology.orchestra.run.smithi079.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 4.5 MB/s | 1.0 MB 00:00 2024-03-17T23:41:39.539 INFO:teuthology.orchestra.run.smithi079.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 329 kB/s | 58 kB 00:00 2024-03-17T23:41:39.590 INFO:teuthology.orchestra.run.smithi079.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-03-17T23:41:39.615 INFO:teuthology.orchestra.run.smithi079.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-17T23:41:39.749 INFO:teuthology.orchestra.run.smithi079.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 567 kB/s | 90 kB 00:00 2024-03-17T23:41:39.774 INFO:teuthology.orchestra.run.smithi079.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 178 kB/s | 28 kB 00:00 2024-03-17T23:41:39.807 INFO:teuthology.orchestra.run.smithi079.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 524 kB/s | 31 kB 00:00 2024-03-17T23:41:39.866 INFO:teuthology.orchestra.run.smithi079.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-03-17T23:41:39.908 INFO:teuthology.orchestra.run.smithi079.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-03-17T23:41:39.950 INFO:teuthology.orchestra.run.smithi079.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 427 kB/s | 36 kB 00:00 2024-03-17T23:41:40.217 INFO:teuthology.orchestra.run.smithi079.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 165 kB/s | 51 kB 00:00 2024-03-17T23:41:40.259 INFO:teuthology.orchestra.run.smithi079.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 744 kB/s | 230 kB 00:00 2024-03-17T23:41:40.293 INFO:teuthology.orchestra.run.smithi079.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-17T23:41:40.377 INFO:teuthology.orchestra.run.smithi079.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-17T23:41:40.410 INFO:teuthology.orchestra.run.smithi079.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 170 kB/s | 20 kB 00:00 2024-03-17T23:41:40.452 INFO:teuthology.orchestra.run.smithi079.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-17T23:41:40.595 INFO:teuthology.orchestra.run.smithi079.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-03-17T23:41:42.828 INFO:teuthology.orchestra.run.smithi079.stdout:(150/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 1.4 MB/s | 3.1 MB 00:02 2024-03-17T23:41:42.870 INFO:teuthology.orchestra.run.smithi079.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.8 MB/s | 321 kB 00:00 2024-03-17T23:41:45.220 INFO:teuthology.orchestra.run.smithi079.stdout:(152/174): librados2-19.0.0-2162.g3d78f5ba.el9. 729 kB/s | 3.4 MB 00:04 2024-03-17T23:41:46.490 INFO:teuthology.orchestra.run.smithi079.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 648 kB/s | 2.3 MB 00:03 2024-03-17T23:41:47.112 INFO:teuthology.orchestra.run.smithi079.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 68 kB/s | 42 kB 00:00 2024-03-17T23:41:48.012 INFO:teuthology.orchestra.run.smithi079.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 276 kB/s | 248 kB 00:00 2024-03-17T23:41:48.104 INFO:teuthology.orchestra.run.smithi079.stdout:(156/174): NetworkManager-libnm-1.46.0-1.el9.x8 645 kB/s | 1.8 MB 00:02 2024-03-17T23:41:48.820 INFO:teuthology.orchestra.run.smithi079.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 88 kB/s | 71 kB 00:00 2024-03-17T23:41:48.863 INFO:teuthology.orchestra.run.smithi079.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 48 kB/s | 36 kB 00:00 2024-03-17T23:41:49.146 INFO:teuthology.orchestra.run.smithi012.stdout:(95/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 1.3 MB/s | 48 MB 00:36 2024-03-17T23:41:49.929 INFO:teuthology.orchestra.run.smithi079.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 173 kB/s | 192 kB 00:01 2024-03-17T23:41:50.029 INFO:teuthology.orchestra.run.smithi079.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 219 kB/s | 256 kB 00:01 2024-03-17T23:41:50.855 INFO:teuthology.orchestra.run.smithi079.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 25 kB/s | 23 kB 00:00 2024-03-17T23:41:50.980 INFO:teuthology.orchestra.run.smithi079.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 114 kB/s | 108 kB 00:00 2024-03-17T23:41:51.614 INFO:teuthology.orchestra.run.smithi079.stdout:(163/174): ceph-test-19.0.0-2162.g3d78f5ba.el9. 1.1 MB/s | 48 MB 00:43 2024-03-17T23:41:51.830 INFO:teuthology.orchestra.run.smithi079.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 159 kB/s | 155 kB 00:00 2024-03-17T23:41:51.981 INFO:teuthology.orchestra.run.smithi079.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 136 kB/s | 136 kB 00:01 2024-03-17T23:41:52.273 INFO:teuthology.orchestra.run.smithi079.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 97 kB/s | 64 kB 00:00 2024-03-17T23:41:52.440 INFO:teuthology.orchestra.run.smithi079.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 47 kB/s | 29 kB 00:00 2024-03-17T23:41:54.273 INFO:teuthology.orchestra.run.smithi079.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 541 kB/s | 1.2 MB 00:02 2024-03-17T23:41:55.615 INFO:teuthology.orchestra.run.smithi079.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 347 kB/s | 465 kB 00:01 2024-03-17T23:41:55.740 INFO:teuthology.orchestra.run.smithi079.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 637 kB/s | 2.2 MB 00:03 2024-03-17T23:41:56.132 INFO:teuthology.orchestra.run.smithi079.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 633 kB/s | 2.3 MB 00:03 2024-03-17T23:41:56.240 INFO:teuthology.orchestra.run.smithi079.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 50 kB/s | 31 kB 00:00 2024-03-17T23:41:56.316 INFO:teuthology.orchestra.run.smithi079.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 406 kB/s | 74 kB 00:00 2024-03-17T23:41:57.857 INFO:teuthology.orchestra.run.smithi079.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 519 kB/s | 1.1 MB 00:02 2024-03-17T23:41:57.864 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-17T23:41:57.864 INFO:teuthology.orchestra.run.smithi079.stdout:Total 3.1 MB/s | 224 MB 01:11 2024-03-17T23:41:58.323 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-17T23:41:58.593 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-17T23:41:58.593 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-17T23:42:00.432 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-17T23:42:00.432 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-17T23:42:07.362 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-17T23:42:07.477 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-17T23:42:07.511 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-17T23:42:07.542 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-17T23:42:07.552 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-17T23:42:07.592 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-17T23:42:07.645 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-17T23:42:07.679 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-17T23:42:07.705 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-17T23:42:07.746 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-17T23:42:08.067 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-17T23:42:08.123 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:42:08.235 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:42:08.258 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:42:08.348 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:42:08.378 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-17T23:42:08.417 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-17T23:42:08.610 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-17T23:42:08.660 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-17T23:42:08.688 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:42:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:42:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-17T23:42:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-17T23:42:08.885 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-17T23:42:08.900 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-17T23:42:08.915 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-17T23:42:09.108 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-17T23:42:09.139 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-17T23:42:09.180 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-17T23:42:09.286 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-17T23:42:09.394 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-17T23:42:09.451 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-17T23:42:09.520 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:42:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:42:09.808 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:42:09.916 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-17T23:42:10.010 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-17T23:42:10.050 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-17T23:42:10.076 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-17T23:42:10.112 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-17T23:42:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-17T23:42:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-17T23:42:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-17T23:42:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-17T23:42:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-17T23:42:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-17T23:42:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-17T23:42:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-17T23:42:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-17T23:42:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-17T23:42:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-17T23:42:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-17T23:42:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-17T23:42:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-17T23:42:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-17T23:42:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-17T23:42:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-17T23:42:11.551 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-17T23:42:11.639 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-17T23:42:11.708 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-17T23:42:11.768 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-17T23:42:11.776 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-17T23:42:11.808 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-17T23:42:11.849 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-17T23:42:11.887 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-17T23:42:11.917 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-17T23:42:12.140 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-17T23:42:12.309 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-17T23:42:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-17T23:42:12.422 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-17T23:42:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-17T23:42:12.604 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-17T23:42:12.659 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-17T23:42:12.739 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-17T23:42:12.772 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-17T23:42:12.809 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-17T23:42:12.855 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-17T23:42:12.891 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-17T23:42:12.934 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-17T23:42:12.976 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-17T23:42:13.109 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-17T23:42:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-17T23:42:13.188 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-17T23:42:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-17T23:42:13.323 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-17T23:42:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-17T23:42:13.720 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-17T23:42:13.746 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-17T23:42:13.770 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-17T23:42:13.834 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-17T23:42:13.971 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-17T23:42:14.069 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-17T23:42:14.113 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-17T23:42:14.161 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-17T23:42:14.216 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-17T23:42:14.408 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-17T23:42:14.450 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-17T23:42:14.578 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-17T23:42:14.616 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-17T23:42:14.648 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-17T23:42:14.679 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-17T23:42:14.732 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-17T23:42:14.767 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-17T23:42:14.806 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-17T23:42:14.838 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-17T23:42:14.868 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-17T23:42:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-17T23:42:14.986 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-17T23:42:15.020 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-17T23:42:15.061 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-17T23:42:15.096 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-17T23:42:15.183 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-17T23:42:15.274 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-17T23:42:15.320 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-17T23:42:15.359 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-17T23:42:15.406 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-17T23:42:15.502 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-17T23:42:15.552 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-17T23:42:16.272 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-17T23:42:16.325 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-17T23:42:16.359 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-17T23:42:16.406 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-17T23:42:16.485 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-17T23:42:16.518 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-17T23:42:16.695 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-17T23:42:16.716 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-17T23:42:16.793 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-17T23:42:17.415 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-17T23:42:17.529 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-17T23:42:18.723 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-17T23:42:18.794 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-17T23:42:18.825 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-17T23:42:18.857 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-17T23:42:18.884 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-17T23:42:19.195 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-17T23:42:19.223 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-17T23:42:19.304 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-17T23:42:19.336 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-17T23:42:19.374 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-17T23:42:19.836 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-17T23:42:19.861 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-17T23:42:19.924 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-17T23:42:19.947 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-17T23:42:22.164 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:42:22.194 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:42:22.732 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:42:22.764 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-17T23:42:22.846 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-17T23:42:22.846 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-17T23:42:22.847 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-17T23:42:22.847 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:42:22.874 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-17T23:42:34.620 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-17T23:42:34.620 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2024-03-17T23:42:34.620 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2024-03-17T23:42:34.620 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2024-03-17T23:42:34.621 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2024-03-17T23:42:34.621 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2024-03-17T23:42:34.621 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /root 2024-03-17T23:42:34.621 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2024-03-17T23:42:34.621 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:42:34.786 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-17T23:42:34.800 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-17T23:42:34.801 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:42:34.801 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-17T23:42:34.801 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:42:36.256 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-17T23:42:36.281 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-17T23:42:36.281 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:42:36.281 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-17T23:42:36.281 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-17T23:42:36.281 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-17T23:42:36.281 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:42:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-17T23:42:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-17T23:42:36.528 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:42:36.529 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-17T23:42:36.529 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-17T23:42:36.529 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-17T23:42:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:42:36.965 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-17T23:42:36.987 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-17T23:42:36.987 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:42:36.987 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-17T23:42:36.987 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-17T23:42:36.987 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-17T23:42:36.988 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:42:37.084 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-17T23:42:37.510 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-17T23:42:37.542 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-17T23:42:37.582 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-17T23:42:37.641 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-17T23:42:37.665 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-17T23:42:37.898 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-17T23:42:37.924 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-17T23:42:38.892 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-17T23:42:38.916 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-17T23:42:39.016 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-17T23:42:39.170 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-17T23:42:39.202 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-17T23:42:39.224 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-17T23:42:39.224 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:42:39.224 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-17T23:42:39.224 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-17T23:42:39.224 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-17T23:42:39.224 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:42:39.269 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-17T23:42:39.280 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-17T23:42:39.325 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-17T23:42:40.196 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-17T23:42:40.230 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-17T23:42:40.253 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-17T23:42:40.253 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:42:40.253 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-17T23:42:40.253 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-17T23:42:40.253 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-17T23:42:40.254 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:42:40.297 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-17T23:42:40.319 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-17T23:42:40.319 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:42:40.319 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-17T23:42:40.319 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:42:40.365 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:42:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:42:40.387 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:42:40.387 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-17T23:42:40.387 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-17T23:42:40.387 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-17T23:42:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:42:40.699 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-17T23:42:40.721 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-17T23:42:40.721 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:42:40.721 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-17T23:42:40.721 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-17T23:42:40.722 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-17T23:42:40.722 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:42:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-17T23:42:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-17T23:42:45.259 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-17T23:42:45.294 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-17T23:42:45.472 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-17T23:42:45.587 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-17T23:42:45.627 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-17T23:42:45.666 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-17T23:42:45.727 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-17T23:42:45.771 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-17T23:42:45.795 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-17T23:42:45.828 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-17T23:42:45.858 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-17T23:42:45.903 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-17T23:42:45.973 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-17T23:42:45.998 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-17T23:42:46.561 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:42:46.605 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:42:46.624 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:42:47.437 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-17T23:42:47.519 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-17T23:42:47.567 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-17T23:42:47.649 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-17T23:42:47.690 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-17T23:42:47.715 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-17T23:42:47.741 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-17T23:42:47.767 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-17T23:42:47.768 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-17T23:42:47.839 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-17T23:42:47.869 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-17T23:42:47.869 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-17T23:42:47.953 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-17T23:42:47.980 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-17T23:42:48.013 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-17T23:42:48.038 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-17T23:42:48.063 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-17T23:42:48.096 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-17T23:42:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-17T23:42:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-17T23:42:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-17T23:42:48.228 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-17T23:42:48.254 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-17T23:42:48.279 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-17T23:42:48.285 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:42:48.285 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:42:48.465 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:42:48.465 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-17T23:42:48.548 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-17T23:42:48.573 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-17T23:42:48.598 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-17T23:42:48.631 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-17T23:42:48.658 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-17T23:42:48.722 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-17T23:42:50.366 INFO:teuthology.orchestra.run.smithi012.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 288 B/s | 21 kB 01:16 2024-03-17T23:42:50.391 INFO:teuthology.orchestra.run.smithi012.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 341 B/s | 25 kB 01:16 2024-03-17T23:42:50.642 INFO:teuthology.orchestra.run.smithi012.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-17T23:42:50.675 INFO:teuthology.orchestra.run.smithi012.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 892 kB/s | 248 kB 00:00 2024-03-17T23:42:50.759 INFO:teuthology.orchestra.run.smithi012.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 423 kB/s | 35 kB 00:00 2024-03-17T23:42:50.818 INFO:teuthology.orchestra.run.smithi012.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.2 MB/s | 216 kB 00:00 2024-03-17T23:42:50.918 INFO:teuthology.orchestra.run.smithi012.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 282 kB/s | 44 kB 00:00 2024-03-17T23:42:51.001 INFO:teuthology.orchestra.run.smithi012.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 514 kB/s | 94 kB 00:00 2024-03-17T23:42:51.027 INFO:teuthology.orchestra.run.smithi012.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 207 kB/s | 22 kB 00:00 2024-03-17T23:42:51.219 INFO:teuthology.orchestra.run.smithi012.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.9 MB/s | 863 kB 00:00 2024-03-17T23:42:51.294 INFO:teuthology.orchestra.run.smithi012.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 68 kB/s | 4.1 MB 01:02 2024-03-17T23:42:51.803 INFO:teuthology.orchestra.run.smithi012.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 39 kB/s | 19 kB 00:00 2024-03-17T23:42:52.070 INFO:teuthology.orchestra.run.smithi012.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 363 kB/s | 309 kB 00:00 2024-03-17T23:42:52.178 INFO:teuthology.orchestra.run.smithi012.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 242 kB/s | 26 kB 00:00 2024-03-17T23:42:52.295 INFO:teuthology.orchestra.run.smithi012.stdout:(110/174): abseil-cpp-20211102.0-3.el9.x86_64.r 432 kB/s | 548 kB 00:01 2024-03-17T23:42:52.320 INFO:teuthology.orchestra.run.smithi012.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 343 kB/s | 49 kB 00:00 2024-03-17T23:42:52.404 INFO:teuthology.orchestra.run.smithi012.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 625 kB/s | 67 kB 00:00 2024-03-17T23:42:52.613 INFO:teuthology.orchestra.run.smithi012.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 518 kB/s | 151 kB 00:00 2024-03-17T23:42:53.363 INFO:teuthology.orchestra.run.smithi012.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 731 kB/s | 548 kB 00:00 2024-03-17T23:42:53.480 INFO:teuthology.orchestra.run.smithi012.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 780 kB/s | 838 kB 00:01 2024-03-17T23:42:53.505 INFO:teuthology.orchestra.run.smithi012.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 311 kB/s | 43 kB 00:00 2024-03-17T23:42:53.589 INFO:teuthology.orchestra.run.smithi012.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 298 kB/s | 32 kB 00:00 2024-03-17T23:42:53.614 INFO:teuthology.orchestra.run.smithi012.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 139 kB/s | 15 kB 00:00 2024-03-17T23:42:53.864 INFO:teuthology.orchestra.run.smithi012.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 626 kB/s | 172 kB 00:00 2024-03-17T23:42:53.986 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-17T23:42:53.987 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-17T23:42:53.987 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-17T23:42:53.987 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-17T23:42:53.987 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-17T23:42:53.987 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-17T23:42:53.987 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-17T23:42:53.987 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-17T23:42:53.987 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-17T23:42:53.987 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-17T23:42:53.987 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:42:53.987 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:42:53.987 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-17T23:42:53.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-17T23:42:53.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-17T23:42:53.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-17T23:42:53.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:42:53.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-17T23:42:53.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-17T23:42:53.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-17T23:42:53.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-17T23:42:53.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-17T23:42:53.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-17T23:42:53.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-17T23:42:53.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-17T23:42:53.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-17T23:42:54.005 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-17T23:42:54.005 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-17T23:42:54.005 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-17T23:42:54.005 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-17T23:42:54.005 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-17T23:42:54.005 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-17T23:42:54.005 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-17T23:42:54.005 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-17T23:42:54.005 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-17T23:42:54.005 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-17T23:42:54.005 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-17T23:42:54.006 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-17T23:42:54.006 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-17T23:42:54.006 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-17T23:42:54.006 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-17T23:42:54.006 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-17T23:42:54.006 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-17T23:42:54.006 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-17T23:42:54.006 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-17T23:42:54.006 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-17T23:42:54.006 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-17T23:42:54.006 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-17T23:42:54.006 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-17T23:42:54.006 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-17T23:42:54.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-17T23:42:54.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-17T23:42:54.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-17T23:42:54.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-17T23:42:54.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-17T23:42:54.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-17T23:42:54.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-17T23:42:54.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-17T23:42:54.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-17T23:42:54.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-17T23:42:54.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-17T23:42:54.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-17T23:42:54.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-17T23:42:54.008 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-17T23:42:54.008 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-17T23:42:54.008 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-17T23:42:54.008 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-17T23:42:54.008 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-17T23:42:54.008 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-17T23:42:54.008 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-17T23:42:54.008 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-17T23:42:54.008 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-17T23:42:54.008 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-17T23:42:54.008 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-17T23:42:54.008 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-17T23:42:54.008 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-17T23:42:54.009 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-17T23:42:54.009 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-17T23:42:54.009 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-17T23:42:54.009 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-17T23:42:54.009 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-17T23:42:54.009 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-17T23:42:54.009 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-17T23:42:54.009 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-17T23:42:54.009 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-17T23:42:54.009 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-17T23:42:54.009 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-17T23:42:54.009 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-17T23:42:54.009 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-17T23:42:54.010 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-17T23:42:54.010 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-17T23:42:54.010 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-17T23:42:54.010 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-17T23:42:54.010 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-17T23:42:54.010 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-17T23:42:54.010 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-17T23:42:54.010 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-17T23:42:54.010 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-17T23:42:54.010 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-17T23:42:54.010 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-17T23:42:54.010 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-17T23:42:54.011 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-17T23:42:54.011 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-17T23:42:54.011 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-17T23:42:54.011 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-17T23:42:54.011 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-17T23:42:54.011 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-17T23:42:54.011 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-17T23:42:54.011 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-17T23:42:54.011 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-17T23:42:54.011 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-17T23:42:54.011 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-17T23:42:54.011 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-17T23:42:54.011 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-17T23:42:54.012 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-17T23:42:54.013 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-17T23:42:54.014 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-17T23:42:54.014 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-17T23:42:54.014 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-17T23:42:54.014 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-17T23:42:54.014 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-17T23:42:54.014 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-17T23:42:54.014 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-17T23:42:54.014 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-17T23:42:54.014 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-17T23:42:54.014 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:42:54.014 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-17T23:42:54.014 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-17T23:42:54.015 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-17T23:42:54.015 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-17T23:42:54.015 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-17T23:42:54.015 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-17T23:42:54.015 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-17T23:42:54.015 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-17T23:42:54.015 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-17T23:42:54.015 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-17T23:42:54.015 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-17T23:42:54.015 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-17T23:42:54.015 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-17T23:42:54.015 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-17T23:42:54.015 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-17T23:42:54.016 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-17T23:42:54.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-17T23:42:54.018 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-17T23:42:54.018 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-17T23:42:54.018 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-17T23:42:54.056 INFO:teuthology.orchestra.run.smithi012.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 812 kB/s | 358 kB 00:00 2024-03-17T23:42:54.181 INFO:teuthology.orchestra.run.smithi012.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 284 kB/s | 35 kB 00:00 2024-03-17T23:42:54.248 INFO:teuthology.orchestra.run.smithi012.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 791 kB/s | 303 kB 00:00 2024-03-17T23:42:54.457 INFO:teuthology.orchestra.run.smithi012.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 738 kB/s | 203 kB 00:00 2024-03-17T23:42:54.732 INFO:teuthology.orchestra.run.smithi012.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 526 kB/s | 144 kB 00:00 2024-03-17T23:42:54.849 INFO:teuthology.orchestra.run.smithi012.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2024-03-17T23:42:54.966 INFO:teuthology.orchestra.run.smithi012.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 153 kB/s | 18 kB 00:00 2024-03-17T23:42:55.075 INFO:teuthology.orchestra.run.smithi012.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 215 kB/s | 23 kB 00:00 2024-03-17T23:42:55.183 INFO:teuthology.orchestra.run.smithi012.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 181 kB/s | 19 kB 00:00 2024-03-17T23:42:55.292 INFO:teuthology.orchestra.run.smithi012.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 183 kB/s | 20 kB 00:00 2024-03-17T23:42:56.717 INFO:teuthology.orchestra.run.smithi012.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 734 kB/s | 1.0 MB 00:01 2024-03-17T23:42:56.826 INFO:teuthology.orchestra.run.smithi012.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 432 kB/s | 46 kB 00:00 2024-03-17T23:42:56.942 INFO:teuthology.orchestra.run.smithi012.stdout:(132/174): python3-grpcio-1.46.7-10.el9.x86_64. 775 kB/s | 2.0 MB 00:02 2024-03-17T23:42:56.968 INFO:teuthology.orchestra.run.smithi012.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 557 kB/s | 79 kB 00:00 2024-03-17T23:42:57.051 INFO:teuthology.orchestra.run.smithi012.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 533 kB/s | 58 kB 00:00 2024-03-17T23:42:57.151 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-17T23:42:57.151 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:42:57.151 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2024-03-17T23:42:57.151 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-17T23:42:57.151 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-17T23:42:57.151 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-17T23:42:57.151 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-17T23:42:57.152 INFO:teuthology.orchestra.run.smithi079.stdout: acl-2.3.1-4.el9.x86_64 2024-03-17T23:42:57.152 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-17T23:42:57.152 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-17T23:42:57.152 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-17T23:42:57.152 INFO:teuthology.orchestra.run.smithi079.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-17T23:42:57.152 INFO:teuthology.orchestra.run.smithi079.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-17T23:42:57.152 INFO:teuthology.orchestra.run.smithi079.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-17T23:42:57.152 INFO:teuthology.orchestra.run.smithi079.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-17T23:42:57.152 INFO:teuthology.orchestra.run.smithi079.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-17T23:42:57.152 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.153 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.153 INFO:teuthology.orchestra.run.smithi079.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-17T23:42:57.153 INFO:teuthology.orchestra.run.smithi079.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-17T23:42:57.153 INFO:teuthology.orchestra.run.smithi079.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-17T23:42:57.153 INFO:teuthology.orchestra.run.smithi079.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-17T23:42:57.153 INFO:teuthology.orchestra.run.smithi079.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-17T23:42:57.153 INFO:teuthology.orchestra.run.smithi079.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-17T23:42:57.153 INFO:teuthology.orchestra.run.smithi079.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-17T23:42:57.153 INFO:teuthology.orchestra.run.smithi079.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-17T23:42:57.153 INFO:teuthology.orchestra.run.smithi079.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-17T23:42:57.154 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-17T23:42:57.154 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-17T23:42:57.154 INFO:teuthology.orchestra.run.smithi079.stdout: automake-1.16.2-8.el9.noarch 2024-03-17T23:42:57.154 INFO:teuthology.orchestra.run.smithi079.stdout: bison-3.7.4-5.el9.x86_64 2024-03-17T23:42:57.154 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-17T23:42:57.154 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.154 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.154 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.154 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.154 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:42:57.154 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.154 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.154 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.155 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:42:57.155 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:42:57.155 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:42:57.155 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:42:57.155 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:42:57.155 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.155 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.155 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:42:57.155 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.155 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.155 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.155 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:42:57.155 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:42:57.156 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.156 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.156 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:42:57.156 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-17T23:42:57.156 INFO:teuthology.orchestra.run.smithi079.stdout: flex-2.6.4-9.el9.x86_64 2024-03-17T23:42:57.156 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-17T23:42:57.156 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-17T23:42:57.156 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-17T23:42:57.156 INFO:teuthology.orchestra.run.smithi079.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-17T23:42:57.156 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-17T23:42:57.156 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-17T23:42:57.156 INFO:teuthology.orchestra.run.smithi079.stdout: jq-1.6-16.el9.x86_64 2024-03-17T23:42:57.156 INFO:teuthology.orchestra.run.smithi079.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-17T23:42:57.157 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-17T23:42:57.157 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-17T23:42:57.157 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.157 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.157 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.157 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-17T23:42:57.157 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-17T23:42:57.157 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-17T23:42:57.157 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-17T23:42:57.157 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-17T23:42:57.157 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.157 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.157 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-17T23:42:57.158 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.158 INFO:teuthology.orchestra.run.smithi079.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-17T23:42:57.158 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-17T23:42:57.158 INFO:teuthology.orchestra.run.smithi079.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-17T23:42:57.158 INFO:teuthology.orchestra.run.smithi079.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-17T23:42:57.158 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-17T23:42:57.158 INFO:teuthology.orchestra.run.smithi079.stdout: lua-5.4.4-4.el9.x86_64 2024-03-17T23:42:57.158 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-17T23:42:57.158 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-17T23:42:57.158 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-17T23:42:57.158 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-17T23:42:57.158 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-13.11-1.el9.x86_64 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-17T23:42:57.159 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.160 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.160 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.160 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-17T23:42:57.160 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-17T23:42:57.160 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-17T23:42:57.160 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-17T23:42:57.160 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-17T23:42:57.160 INFO:teuthology.orchestra.run.smithi079.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-17T23:42:57.160 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-17T23:42:57.160 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-17T23:42:57.160 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-17T23:42:57.160 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-17T23:42:57.160 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-17T23:42:57.161 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-17T23:42:57.162 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-17T23:42:57.162 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-17T23:42:57.162 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-17T23:42:57.162 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-17T23:42:57.162 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply-3.11-14.el9.noarch 2024-03-17T23:42:57.162 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-17T23:42:57.162 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-17T23:42:57.162 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-17T23:42:57.162 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-17T23:42:57.162 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-17T23:42:57.162 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-17T23:42:57.163 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-17T23:42:57.164 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-17T23:42:57.164 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-17T23:42:57.164 INFO:teuthology.orchestra.run.smithi079.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-17T23:42:57.164 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-17T23:42:57.164 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-17T23:42:57.164 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-17T23:42:57.164 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-17T23:42:57.164 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-17T23:42:57.164 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-17T23:42:57.164 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-17T23:42:57.164 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.164 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.164 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:42:57.165 INFO:teuthology.orchestra.run.smithi079.stdout: re2-1:20211101-3.el9.x86_64 2024-03-17T23:42:57.165 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-17T23:42:57.165 INFO:teuthology.orchestra.run.smithi079.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-17T23:42:57.165 INFO:teuthology.orchestra.run.smithi079.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-17T23:42:57.165 INFO:teuthology.orchestra.run.smithi079.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-17T23:42:57.165 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-17T23:42:57.166 INFO:teuthology.orchestra.run.smithi012.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 153 kB/s | 16 kB 00:00 2024-03-17T23:42:57.166 INFO:teuthology.orchestra.run.smithi079.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-17T23:42:57.166 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-17T23:42:57.166 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-17T23:42:57.335 INFO:teuthology.orchestra.run.smithi012.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 743 kB/s | 272 kB 00:00 2024-03-17T23:42:57.391 INFO:teuthology.orchestra.run.smithi012.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 391 kB/s | 90 kB 00:00 2024-03-17T23:42:57.463 INFO:teuthology.orchestra.run.smithi012.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 223 kB/s | 28 kB 00:00 2024-03-17T23:42:57.580 INFO:teuthology.orchestra.run.smithi012.stdout:(139/174): libarrow-9.0.0-10.el9.x86_64.rpm 783 kB/s | 4.4 MB 00:05 2024-03-17T23:42:57.605 INFO:teuthology.orchestra.run.smithi012.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 144 kB/s | 31 kB 00:00 2024-03-17T23:42:57.649 DEBUG:teuthology.parallel:result is None 2024-03-17T23:42:57.705 INFO:teuthology.orchestra.run.smithi012.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 474 kB/s | 59 kB 00:00 2024-03-17T23:42:57.731 INFO:teuthology.orchestra.run.smithi012.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 286 kB/s | 36 kB 00:00 2024-03-17T23:42:57.798 INFO:teuthology.orchestra.run.smithi012.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 564 kB/s | 188 kB 00:00 2024-03-17T23:42:57.832 INFO:teuthology.orchestra.run.smithi012.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 406 kB/s | 51 kB 00:00 2024-03-17T23:42:58.024 INFO:teuthology.orchestra.run.smithi012.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 399 kB/s | 90 kB 00:00 2024-03-17T23:42:58.157 INFO:teuthology.orchestra.run.smithi012.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-03-17T23:42:58.182 INFO:teuthology.orchestra.run.smithi012.stdout:(147/174): python3-webob-1.8.7-6.el9.noarch.rpm 510 kB/s | 230 kB 00:00 2024-03-17T23:42:58.483 INFO:teuthology.orchestra.run.smithi012.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 591 kB/s | 192 kB 00:00 2024-03-17T23:42:58.616 INFO:teuthology.orchestra.run.smithi012.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 545 kB/s | 427 kB 00:00 2024-03-17T23:42:59.017 INFO:teuthology.orchestra.run.smithi012.stdout:(150/174): librados2-19.0.0-2162.g3d78f5ba.el9. 6.4 MB/s | 3.4 MB 00:00 2024-03-17T23:42:59.067 INFO:teuthology.orchestra.run.smithi012.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2024-03-17T23:42:59.925 INFO:teuthology.orchestra.run.smithi012.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 2.7 MB/s | 2.3 MB 00:00 2024-03-17T23:43:00.126 INFO:teuthology.orchestra.run.smithi012.stdout:(153/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 2.1 MB/s | 3.1 MB 00:01 2024-03-17T23:43:00.292 INFO:teuthology.orchestra.run.smithi012.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 256 kB/s | 42 kB 00:00 2024-03-17T23:43:00.484 INFO:teuthology.orchestra.run.smithi012.stdout:(155/174): thrift-0.15.0-2.el9.x86_64.rpm 704 kB/s | 1.6 MB 00:02 2024-03-17T23:43:00.518 INFO:teuthology.orchestra.run.smithi012.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.1 MB/s | 248 kB 00:00 2024-03-17T23:43:00.585 INFO:teuthology.orchestra.run.smithi012.stdout:(157/174): NetworkManager-libnm-1.46.0-1.el9.x8 2.8 MB/s | 1.8 MB 00:00 2024-03-17T23:43:00.610 INFO:teuthology.orchestra.run.smithi012.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 399 kB/s | 36 kB 00:00 2024-03-17T23:43:00.710 INFO:teuthology.orchestra.run.smithi012.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.5 MB/s | 192 kB 00:00 2024-03-17T23:43:00.744 INFO:teuthology.orchestra.run.smithi012.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.9 MB/s | 256 kB 00:00 2024-03-17T23:43:00.777 INFO:teuthology.orchestra.run.smithi012.stdout:(161/174): acl-2.3.1-4.el9.x86_64.rpm 244 kB/s | 71 kB 00:00 2024-03-17T23:43:00.803 INFO:teuthology.orchestra.run.smithi012.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 252 kB/s | 23 kB 00:00 2024-03-17T23:43:00.829 INFO:teuthology.orchestra.run.smithi012.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-17T23:43:00.878 INFO:teuthology.orchestra.run.smithi012.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-17T23:43:00.904 INFO:teuthology.orchestra.run.smithi012.stdout:(165/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 155 kB 00:00 2024-03-17T23:43:00.929 INFO:teuthology.orchestra.run.smithi012.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 641 kB/s | 64 kB 00:00 2024-03-17T23:43:00.954 INFO:teuthology.orchestra.run.smithi012.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 385 kB/s | 29 kB 00:00 2024-03-17T23:43:01.346 INFO:teuthology.orchestra.run.smithi012.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 2.9 MB/s | 1.2 MB 00:00 2024-03-17T23:43:01.555 INFO:teuthology.orchestra.run.smithi012.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 2.2 MB/s | 465 kB 00:00 2024-03-17T23:43:01.613 INFO:teuthology.orchestra.run.smithi012.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 3.5 MB/s | 2.3 MB 00:00 2024-03-17T23:43:01.638 INFO:teuthology.orchestra.run.smithi012.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 376 kB/s | 31 kB 00:00 2024-03-17T23:43:01.714 INFO:teuthology.orchestra.run.smithi012.stdout:(172/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 2.8 MB/s | 2.2 MB 00:00 2024-03-17T23:43:01.764 INFO:teuthology.orchestra.run.smithi012.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 596 kB/s | 74 kB 00:00 2024-03-17T23:43:01.939 INFO:teuthology.orchestra.run.smithi012.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 3.3 MB/s | 1.1 MB 00:00 2024-03-17T23:43:01.952 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-17T23:43:01.953 INFO:teuthology.orchestra.run.smithi012.stdout:Total 1.7 MB/s | 224 MB 02:14 2024-03-17T23:43:03.476 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-17T23:43:03.876 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-17T23:43:03.876 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-17T23:43:06.371 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-17T23:43:06.371 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-17T23:43:13.777 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-17T23:43:13.875 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-17T23:43:13.919 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-17T23:43:13.965 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-17T23:43:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-17T23:43:14.108 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-17T23:43:14.215 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-17T23:43:14.278 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-17T23:43:14.317 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-17T23:43:14.417 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-17T23:43:14.838 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-17T23:43:14.874 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:43:15.016 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:43:15.050 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:43:15.173 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:43:15.215 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-17T23:43:15.278 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-17T23:43:15.573 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-17T23:43:15.631 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-17T23:43:15.649 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:43:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:43:15.816 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-17T23:43:15.918 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-17T23:43:15.985 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-17T23:43:16.010 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-17T23:43:16.074 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-17T23:43:16.365 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-17T23:43:16.411 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-17T23:43:16.444 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-17T23:43:16.778 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-17T23:43:16.953 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-17T23:43:17.049 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-17T23:43:17.277 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:43:17.328 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:43:17.689 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:43:17.883 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-17T23:43:18.031 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-17T23:43:18.068 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-17T23:43:18.105 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-17T23:43:18.167 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-17T23:43:18.275 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-17T23:43:18.351 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-17T23:43:18.401 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-17T23:43:18.495 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-17T23:43:18.539 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-17T23:43:18.602 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-17T23:43:18.643 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-17T23:43:18.676 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-17T23:43:18.779 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-17T23:43:18.813 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-17T23:43:18.865 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-17T23:43:18.894 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-17T23:43:19.009 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-17T23:43:19.075 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-17T23:43:19.199 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-17T23:43:19.387 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-17T23:43:19.451 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-17T23:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-17T23:43:20.269 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-17T23:43:20.373 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-17T23:43:20.431 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-17T23:43:20.458 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-17T23:43:20.506 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-17T23:43:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-17T23:43:20.641 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-17T23:43:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-17T23:43:21.085 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-17T23:43:21.427 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-17T23:43:21.496 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-17T23:43:21.560 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-17T23:43:21.777 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-17T23:43:21.861 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-17T23:43:21.947 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-17T23:43:22.053 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-17T23:43:22.090 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-17T23:43:22.130 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-17T23:43:22.200 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-17T23:43:22.239 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-17T23:43:22.293 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-17T23:43:22.358 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-17T23:43:22.553 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-17T23:43:22.601 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-17T23:43:22.640 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-17T23:43:22.698 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-17T23:43:22.839 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-17T23:43:22.880 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-17T23:43:22.926 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-17T23:43:22.962 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-17T23:43:22.995 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-17T23:43:23.150 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-17T23:43:23.336 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-17T23:43:23.475 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-17T23:43:23.522 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-17T23:43:23.585 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-17T23:43:23.660 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-17T23:43:23.912 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-17T23:43:23.968 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-17T23:43:24.154 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-17T23:43:24.207 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-17T23:43:24.247 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-17T23:43:24.285 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-17T23:43:24.326 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-17T23:43:24.373 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-17T23:43:24.440 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-17T23:43:24.499 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-17T23:43:24.548 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-17T23:43:24.679 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-17T23:43:24.727 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-17T23:43:24.779 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-17T23:43:24.847 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-17T23:43:24.890 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-17T23:43:25.031 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-17T23:43:25.144 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-17T23:43:25.202 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-17T23:43:25.238 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-17T23:43:25.295 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-17T23:43:25.447 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-17T23:43:25.519 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-17T23:43:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-17T23:43:26.586 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-17T23:43:26.619 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-17T23:43:26.676 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-17T23:43:26.790 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-17T23:43:26.837 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-17T23:43:27.084 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-17T23:43:27.116 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-17T23:43:27.215 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-17T23:43:28.054 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-17T23:43:28.182 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-17T23:43:29.802 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-17T23:43:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-17T23:43:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-17T23:43:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-17T23:43:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-17T23:43:30.413 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-17T23:43:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-17T23:43:30.549 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-17T23:43:30.614 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-17T23:43:30.667 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-17T23:43:31.219 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-17T23:43:31.247 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-17T23:43:31.346 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-17T23:43:31.373 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-17T23:43:34.112 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:43:34.162 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:43:34.828 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:43:34.873 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-17T23:43:35.029 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-17T23:43:35.029 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-17T23:43:35.029 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-17T23:43:35.029 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:43:35.074 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-17T23:43:48.829 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-17T23:43:48.830 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /sys 2024-03-17T23:43:48.830 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /proc 2024-03-17T23:43:48.830 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /mnt 2024-03-17T23:43:48.830 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /var/tmp 2024-03-17T23:43:48.830 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /home 2024-03-17T23:43:48.830 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /root 2024-03-17T23:43:48.830 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /tmp 2024-03-17T23:43:48.830 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:43:49.060 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-17T23:43:49.112 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-17T23:43:49.112 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:43:49.112 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-17T23:43:49.112 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:43:50.798 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-17T23:43:50.898 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-17T23:43:50.898 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:43:50.898 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-17T23:43:50.898 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-17T23:43:50.898 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-17T23:43:50.899 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:43:51.156 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-17T23:43:51.227 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-17T23:43:51.227 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:43:51.227 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-17T23:43:51.227 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-17T23:43:51.227 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-17T23:43:51.227 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:43:51.740 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-17T23:43:51.812 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-17T23:43:51.812 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:43:51.812 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-17T23:43:51.812 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-17T23:43:51.812 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-17T23:43:51.813 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:43:51.934 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-17T23:43:52.551 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-17T23:43:52.584 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-17T23:43:52.637 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-17T23:43:52.741 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-17T23:43:52.771 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-17T23:43:53.120 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-17T23:43:53.155 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-17T23:43:54.332 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-17T23:43:54.364 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-17T23:43:54.515 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-17T23:43:54.695 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-17T23:43:54.724 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-17T23:43:54.810 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-17T23:43:54.811 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:43:54.811 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-17T23:43:54.811 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-17T23:43:54.811 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-17T23:43:54.811 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:43:54.867 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-17T23:43:54.906 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-17T23:43:54.964 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-17T23:43:55.957 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-17T23:43:56.002 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-17T23:43:56.088 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-17T23:43:56.088 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:43:56.088 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-17T23:43:56.088 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-17T23:43:56.088 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-17T23:43:56.088 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:43:56.136 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-17T23:43:56.233 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-17T23:43:56.233 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:43:56.233 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-17T23:43:56.233 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:43:56.320 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:43:56.401 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:43:56.401 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:43:56.401 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-17T23:43:56.401 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-17T23:43:56.401 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-17T23:43:56.401 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:43:56.804 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-17T23:43:56.901 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-17T23:43:56.901 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:43:56.901 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-17T23:43:56.902 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-17T23:43:56.902 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-17T23:43:56.902 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:44:02.018 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-17T23:44:02.067 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-17T23:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-17T23:44:02.137 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-17T23:44:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-17T23:44:02.541 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-17T23:44:02.594 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-17T23:44:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-17T23:44:02.717 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-17T23:44:02.770 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-17T23:44:02.797 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-17T23:44:02.850 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-17T23:44:02.905 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-17T23:44:03.002 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-17T23:44:03.125 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-17T23:44:03.160 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-17T23:44:04.128 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:44:04.174 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:44:04.240 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:44:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-17T23:44:05.770 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-17T23:44:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-17T23:44:05.956 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-17T23:44:05.990 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-17T23:44:06.023 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-17T23:44:06.057 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-17T23:44:06.098 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-17T23:44:06.099 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-17T23:44:06.245 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-17T23:44:06.295 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-17T23:44:06.295 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-17T23:44:06.457 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-17T23:44:06.510 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-17T23:44:06.556 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-17T23:44:06.590 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-17T23:44:06.622 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-17T23:44:06.705 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-17T23:44:06.780 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-17T23:44:06.826 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-17T23:44:06.860 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-17T23:44:06.894 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-17T23:44:06.927 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-17T23:44:06.960 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-17T23:44:06.999 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:44:06.999 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:44:07.344 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:44:07.345 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-17T23:44:07.500 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-17T23:44:07.547 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-17T23:44:07.580 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-17T23:44:07.638 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-17T23:44:07.673 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-17T23:44:07.796 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-17T23:44:08.051 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-17T23:44:08.051 INFO:teuthology.orchestra.run.smithi012.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-17T23:44:08.051 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:44:12.896 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-17T23:44:12.896 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-17T23:44:12.896 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-17T23:44:12.896 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-17T23:44:12.896 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-17T23:44:12.896 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-17T23:44:12.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-17T23:44:12.898 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-17T23:44:12.898 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-17T23:44:12.898 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-17T23:44:12.898 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-17T23:44:12.898 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-17T23:44:12.898 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-17T23:44:12.914 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-17T23:44:12.914 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-17T23:44:12.914 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-17T23:44:12.914 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-17T23:44:12.914 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-17T23:44:12.914 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-17T23:44:12.914 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-17T23:44:12.914 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-17T23:44:12.915 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-17T23:44:12.915 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-17T23:44:12.915 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-17T23:44:12.915 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-17T23:44:12.915 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-17T23:44:12.915 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-17T23:44:12.915 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-17T23:44:12.915 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-17T23:44:12.915 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-17T23:44:12.915 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-17T23:44:12.915 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-17T23:44:12.915 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-17T23:44:12.915 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-17T23:44:12.916 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-17T23:44:12.916 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-17T23:44:12.916 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-17T23:44:12.916 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-17T23:44:12.916 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-17T23:44:12.916 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-17T23:44:12.916 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-17T23:44:12.916 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-17T23:44:12.916 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-17T23:44:12.916 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-17T23:44:12.916 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-17T23:44:12.916 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-17T23:44:12.916 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-17T23:44:12.917 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-17T23:44:12.917 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-17T23:44:12.917 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-17T23:44:12.917 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-17T23:44:12.917 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-17T23:44:12.917 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-17T23:44:12.917 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-17T23:44:12.917 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-17T23:44:12.917 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-17T23:44:12.917 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-17T23:44:12.917 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-17T23:44:12.917 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-17T23:44:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-17T23:44:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-17T23:44:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-17T23:44:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-17T23:44:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-17T23:44:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-17T23:44:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-17T23:44:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-17T23:44:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-17T23:44:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-17T23:44:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-17T23:44:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-17T23:44:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-17T23:44:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-17T23:44:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-17T23:44:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-17T23:44:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-17T23:44:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-17T23:44:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-17T23:44:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-17T23:44:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-17T23:44:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-17T23:44:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-17T23:44:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-17T23:44:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-17T23:44:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-17T23:44:12.921 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-17T23:44:12.921 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-17T23:44:12.921 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-17T23:44:12.921 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-17T23:44:12.921 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-17T23:44:12.921 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-17T23:44:12.921 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-17T23:44:12.921 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-17T23:44:12.921 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-17T23:44:12.921 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-17T23:44:12.921 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-17T23:44:12.921 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-17T23:44:12.921 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-17T23:44:12.922 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-17T23:44:12.922 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-17T23:44:12.922 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-17T23:44:12.922 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-17T23:44:12.922 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-17T23:44:12.922 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-17T23:44:12.922 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-17T23:44:12.922 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-17T23:44:12.922 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-17T23:44:12.922 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-17T23:44:12.922 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-17T23:44:12.922 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-17T23:44:12.923 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-17T23:44:12.924 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-17T23:44:12.925 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-17T23:44:12.926 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-17T23:44:12.926 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-17T23:44:12.926 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-17T23:44:12.926 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-17T23:44:12.926 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-17T23:44:12.926 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-17T23:44:12.926 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-17T23:44:12.926 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-17T23:44:12.926 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-17T23:44:12.926 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-17T23:44:12.926 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-17T23:44:12.926 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-17T23:44:12.926 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-17T23:44:12.927 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-17T23:44:12.927 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-17T23:44:12.927 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-17T23:44:12.927 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-17T23:44:15.664 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-17T23:44:15.664 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:44:15.665 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2024-03-17T23:44:15.665 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-17T23:44:15.665 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-17T23:44:15.665 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-17T23:44:15.665 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-17T23:44:15.665 INFO:teuthology.orchestra.run.smithi012.stdout: acl-2.3.1-4.el9.x86_64 2024-03-17T23:44:15.665 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-17T23:44:15.665 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-17T23:44:15.665 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-17T23:44:15.665 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-17T23:44:15.665 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-17T23:44:15.665 INFO:teuthology.orchestra.run.smithi012.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-17T23:44:15.665 INFO:teuthology.orchestra.run.smithi012.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-17T23:44:15.666 INFO:teuthology.orchestra.run.smithi012.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-17T23:44:15.666 INFO:teuthology.orchestra.run.smithi012.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.666 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.666 INFO:teuthology.orchestra.run.smithi012.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-17T23:44:15.666 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-17T23:44:15.666 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-17T23:44:15.666 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-17T23:44:15.666 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-17T23:44:15.666 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-17T23:44:15.666 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-17T23:44:15.666 INFO:teuthology.orchestra.run.smithi012.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-17T23:44:15.666 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-17T23:44:15.666 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-17T23:44:15.667 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-17T23:44:15.667 INFO:teuthology.orchestra.run.smithi012.stdout: automake-1.16.2-8.el9.noarch 2024-03-17T23:44:15.667 INFO:teuthology.orchestra.run.smithi012.stdout: bison-3.7.4-5.el9.x86_64 2024-03-17T23:44:15.667 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-17T23:44:15.667 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.667 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.667 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.667 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.667 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:44:15.667 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.667 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.667 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.667 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.668 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.669 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:44:15.669 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-17T23:44:15.669 INFO:teuthology.orchestra.run.smithi012.stdout: flex-2.6.4-9.el9.x86_64 2024-03-17T23:44:15.669 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-17T23:44:15.669 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-17T23:44:15.669 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-17T23:44:15.669 INFO:teuthology.orchestra.run.smithi012.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-17T23:44:15.669 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-17T23:44:15.669 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-17T23:44:15.669 INFO:teuthology.orchestra.run.smithi012.stdout: jq-1.6-16.el9.x86_64 2024-03-17T23:44:15.669 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-17T23:44:15.669 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-17T23:44:15.669 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-17T23:44:15.670 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.670 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.670 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.670 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-17T23:44:15.670 INFO:teuthology.orchestra.run.smithi012.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-17T23:44:15.670 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-17T23:44:15.670 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-17T23:44:15.670 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-17T23:44:15.670 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.670 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.670 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-17T23:44:15.670 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.671 INFO:teuthology.orchestra.run.smithi012.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-17T23:44:15.671 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-17T23:44:15.671 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-17T23:44:15.671 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-17T23:44:15.671 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-17T23:44:15.671 INFO:teuthology.orchestra.run.smithi012.stdout: lua-5.4.4-4.el9.x86_64 2024-03-17T23:44:15.671 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-17T23:44:15.671 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-17T23:44:15.671 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-17T23:44:15.671 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-17T23:44:15.671 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-17T23:44:15.671 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-17T23:44:15.671 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-13.11-1.el9.x86_64 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.672 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-17T23:44:15.673 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-17T23:44:15.674 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-17T23:44:15.674 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-17T23:44:15.674 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-17T23:44:15.674 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-17T23:44:15.674 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-17T23:44:15.674 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-17T23:44:15.674 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-17T23:44:15.674 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-17T23:44:15.674 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-17T23:44:15.674 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-17T23:44:15.674 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-17T23:44:15.674 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply-3.11-14.el9.noarch 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-17T23:44:15.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-17T23:44:15.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: re2-1:20211101-3.el9.x86_64 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-17T23:44:15.677 INFO:teuthology.orchestra.run.smithi012.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-17T23:44:15.678 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-17T23:44:15.678 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-17T23:44:15.678 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-17T23:44:15.678 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:44:15.678 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-17T23:44:16.287 DEBUG:teuthology.parallel:result is None 2024-03-17T23:44:16.288 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:44:16.505 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-17T23:44:16.559 INFO:teuthology.orchestra.run.smithi012.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-17T23:44:16.560 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-17T23:44:16.560 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-17T23:44:16.562 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:44:16.707 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-17T23:44:16.732 INFO:teuthology.orchestra.run.smithi079.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-17T23:44:16.732 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-17T23:44:16.732 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-17T23:44:16.734 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:44:16.873 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-17T23:44:16.900 INFO:teuthology.orchestra.run.smithi191.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-17T23:44:16.900 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-17T23:44:16.900 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-17T23:44:16.902 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-17T23:44:16.902 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:44:16.903 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-17T23:44:16.984 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-17T23:44:16.984 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-17T23:44:17.018 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-17T23:44:17.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-17T23:44:17.053 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-17T23:44:17.053 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:44:17.053 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2024-03-17T23:44:17.147 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-17T23:44:17.241 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-17T23:44:17.241 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-03-17T23:44:17.273 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-17T23:44:17.344 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-17T23:44:17.344 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-03-17T23:44:17.375 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-17T23:44:17.447 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-17T23:44:17.448 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:44:17.448 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-17T23:44:17.535 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-17T23:44:17.667 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-17T23:44:17.667 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-17T23:44:17.694 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-17T23:44:17.761 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-17T23:44:17.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-17T23:44:17.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-17T23:44:17.865 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-17T23:44:17.865 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:44:17.865 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2024-03-17T23:44:17.945 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-17T23:44:18.056 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-17T23:44:18.057 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-03-17T23:44:18.083 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-17T23:44:18.150 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-17T23:44:18.150 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-03-17T23:44:18.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-17T23:44:18.253 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-17T23:44:18.363 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-17T23:44:18.363 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:44:18.363 INFO:tasks.cephadm:Cluster fsid is 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:44:18.363 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-17T23:44:18.364 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.12', 'mon.b': '172.21.15.79', 'mon.c': '172.21.15.191'} 2024-03-17T23:44:18.364 INFO:tasks.cephadm:First mon is mon.a on smithi012 2024-03-17T23:44:18.364 INFO:tasks.cephadm:First mgr is x 2024-03-17T23:44:18.364 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-17T23:44:18.364 DEBUG:teuthology.orchestra.run.smithi012:> sudo hostname $(hostname -s) 2024-03-17T23:44:18.448 DEBUG:teuthology.orchestra.run.smithi079:> sudo hostname $(hostname -s) 2024-03-17T23:44:18.475 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname $(hostname -s) 2024-03-17T23:44:18.507 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-17T23:44:18.507 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:44:18.649 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77658/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2162-g3d78f5ba', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2162.g3d78f5ba'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 01:33:04.210510', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240314.183243', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-17T23:44:18.785 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-batrick-testing-20240314.183243, sha1 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:44:18.786 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm 2024-03-17T23:44:18.786 DEBUG:teuthology.orchestra.run.smithi012:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-17T23:44:19.248 INFO:teuthology.orchestra.run.smithi012.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 17 23:44 /home/ubuntu/cephtest/cephadm 2024-03-17T23:44:19.248 DEBUG:teuthology.orchestra.run.smithi079:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-17T23:44:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 17 23:44 /home/ubuntu/cephtest/cephadm 2024-03-17T23:44:19.805 DEBUG:teuthology.orchestra.run.smithi191:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-17T23:44:20.372 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 17 23:44 /home/ubuntu/cephtest/cephadm 2024-03-17T23:44:20.372 DEBUG:teuthology.orchestra.run.smithi012:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-17T23:44:20.435 DEBUG:teuthology.orchestra.run.smithi079:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-17T23:44:20.454 DEBUG:teuthology.orchestra.run.smithi191:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-17T23:44:20.489 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af on all hosts... 2024-03-17T23:44:20.490 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-17T23:44:20.497 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-17T23:44:20.499 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-17T23:44:21.333 INFO:teuthology.orchestra.run.smithi191.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-17T23:44:21.408 INFO:teuthology.orchestra.run.smithi079.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-17T23:44:21.880 INFO:teuthology.orchestra.run.smithi012.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-17T23:44:53.887 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-03-17T23:44:53.887 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:44:53.887 INFO:teuthology.orchestra.run.smithi079.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-17T23:44:53.887 INFO:teuthology.orchestra.run.smithi079.stdout: "repo_digests": [ 2024-03-17T23:44:53.887 INFO:teuthology.orchestra.run.smithi079.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-17T23:44:53.887 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-03-17T23:44:53.887 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-03-17T23:45:25.562 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-17T23:45:25.562 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:45:25.562 INFO:teuthology.orchestra.run.smithi191.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-17T23:45:25.562 INFO:teuthology.orchestra.run.smithi191.stdout: "repo_digests": [ 2024-03-17T23:45:25.563 INFO:teuthology.orchestra.run.smithi191.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-17T23:45:25.563 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-17T23:45:25.563 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-17T23:45:26.147 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-17T23:45:26.147 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:45:26.147 INFO:teuthology.orchestra.run.smithi012.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-17T23:45:26.148 INFO:teuthology.orchestra.run.smithi012.stdout: "repo_digests": [ 2024-03-17T23:45:26.148 INFO:teuthology.orchestra.run.smithi012.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-17T23:45:26.148 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-17T23:45:26.148 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-17T23:45:26.195 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph 2024-03-17T23:45:26.295 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph 2024-03-17T23:45:26.331 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph 2024-03-17T23:45:26.362 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /etc/ceph 2024-03-17T23:45:26.470 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /etc/ceph 2024-03-17T23:45:26.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /etc/ceph 2024-03-17T23:45:26.529 INFO:tasks.cephadm:Writing seed config... 2024-03-17T23:45:26.530 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-17T23:45:26.530 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-17T23:45:26.530 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-17T23:45:26.530 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-17T23:45:26.530 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-17T23:45:26.530 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-17T23:45:26.530 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-17T23:45:26.531 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-17T23:45:26.532 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-17T23:45:26.533 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-17T23:45:26.533 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-17T23:45:26.533 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-17T23:45:26.533 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-17T23:45:26.533 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-17T23:45:26.533 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-17T23:45:26.533 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-17T23:45:26.534 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:45:26.534 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-17T23:45:26.593 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 4558f3ea-e4b8-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-17T23:45:26.593 DEBUG:teuthology.orchestra.run.smithi012:mon.a> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.a.service 2024-03-17T23:45:26.641 DEBUG:teuthology.orchestra.run.smithi012:mgr.x> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mgr.x.service 2024-03-17T23:45:26.687 INFO:tasks.cephadm:Bootstrapping... 2024-03-17T23:45:26.687 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -v bootstrap --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.12 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-17T23:45:26.973 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-17T23:45:26.973 INFO:teuthology.orchestra.run.smithi012.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', '-v', 'bootstrap', '--fsid', '4558f3ea-e4b8-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.12', '--skip-admin-label'] 2024-03-17T23:45:27.032 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-17T23:45:27.032 INFO:teuthology.orchestra.run.smithi012.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-17T23:45:27.033 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying podman|docker is present... 2024-03-17T23:45:27.082 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-17T23:45:27.082 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying lvm2 is present... 2024-03-17T23:45:27.082 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying time synchronization is in place... 2024-03-17T23:45:27.097 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-17T23:45:27.097 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-17T23:45:27.110 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-17T23:45:27.110 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-17T23:45:27.124 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout enabled 2024-03-17T23:45:27.142 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout active 2024-03-17T23:45:27.142 INFO:teuthology.orchestra.run.smithi012.stdout:Unit chronyd.service is enabled and running 2024-03-17T23:45:27.142 INFO:teuthology.orchestra.run.smithi012.stdout:Repeating the final host check... 2024-03-17T23:45:27.199 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-17T23:45:27.199 INFO:teuthology.orchestra.run.smithi012.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-17T23:45:27.199 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl is present 2024-03-17T23:45:27.199 INFO:teuthology.orchestra.run.smithi012.stdout:lvcreate is present 2024-03-17T23:45:27.213 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-17T23:45:27.213 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-17T23:45:27.230 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-17T23:45:27.231 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-17T23:45:27.252 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout enabled 2024-03-17T23:45:27.274 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout active 2024-03-17T23:45:27.274 INFO:teuthology.orchestra.run.smithi012.stdout:Unit chronyd.service is enabled and running 2024-03-17T23:45:27.274 INFO:teuthology.orchestra.run.smithi012.stdout:Host looks OK 2024-03-17T23:45:27.275 INFO:teuthology.orchestra.run.smithi012.stdout:Cluster fsid: 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:45:27.275 INFO:teuthology.orchestra.run.smithi012.stdout:Acquiring lock 140336607416224 on /run/cephadm/4558f3ea-e4b8-11ee-95c9-87774f69a715.lock 2024-03-17T23:45:27.276 INFO:teuthology.orchestra.run.smithi012.stdout:Lock 140336607416224 acquired on /run/cephadm/4558f3ea-e4b8-11ee-95c9-87774f69a715.lock 2024-03-17T23:45:27.276 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying IP 172.21.15.12 port 3300 ... 2024-03-17T23:45:27.278 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying IP 172.21.15.12 port 6789 ... 2024-03-17T23:45:27.278 INFO:teuthology.orchestra.run.smithi012.stdout:Base mon IP(s) is [172.21.15.12:3300, 172.21.15.12:6789], mon addrv is [v2:172.21.15.12:3300,v1:172.21.15.12:6789] 2024-03-17T23:45:27.293 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.12 metric 100 2024-03-17T23:45:27.294 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.12 metric 100 2024-03-17T23:45:27.306 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-17T23:45:27.306 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-17T23:45:27.306 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-17T23:45:27.317 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-17T23:45:27.317 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-17T23:45:27.318 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-17T23:45:27.318 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-17T23:45:27.318 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-03-17T23:45:27.318 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-17T23:45:27.321 INFO:teuthology.orchestra.run.smithi012.stdout:Mon IP `172.21.15.12` is in CIDR network `172.21.0.0/20` 2024-03-17T23:45:27.321 INFO:teuthology.orchestra.run.smithi012.stdout:Mon IP `172.21.15.12` is in CIDR network `172.21.0.0/20` 2024-03-17T23:45:27.321 INFO:teuthology.orchestra.run.smithi012.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-17T23:45:27.324 INFO:teuthology.orchestra.run.smithi012.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-17T23:45:27.326 INFO:teuthology.orchestra.run.smithi012.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-17T23:45:28.135 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-17T23:45:28.136 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-17T23:45:28.136 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Getting image source signatures 2024-03-17T23:45:28.136 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying blob sha256:66487374cb611c95435850c17b53ebb64c22ed9555f72cd43d678b60262a12b1 2024-03-17T23:45:28.136 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-17T23:45:28.136 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying config sha256:be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-17T23:45:28.136 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-17T23:45:29.358 INFO:teuthology.orchestra.run.smithi012.stdout:ceph: stdout ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-17T23:45:29.359 INFO:teuthology.orchestra.run.smithi012.stdout:Ceph version: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-17T23:45:29.359 INFO:teuthology.orchestra.run.smithi012.stdout:Extracting ceph user uid/gid from container image... 2024-03-17T23:45:30.986 INFO:teuthology.orchestra.run.smithi012.stdout:stat: stdout 167 167 2024-03-17T23:45:30.986 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial keys... 2024-03-17T23:45:32.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQCbgPdl8hUZGxAAhseFwCmxd0K3fzkKrW+CUw== 2024-03-17T23:45:33.211 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQCcgPdl0kzYIRAARTSA/+FYmXvSqNKPd4Hs6g== 2024-03-17T23:45:34.362 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQCdgPdlsSxVKxAASn0C5LrSiyhzZymw2LZxUg== 2024-03-17T23:45:34.364 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial monmap... 2024-03-17T23:45:35.498 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-17T23:45:35.498 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-17T23:45:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:45:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-17T23:45:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool for a [v2:172.21.15.12:3300,v1:172.21.15.12:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-17T23:45:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:setting min_mon_release = quincy 2024-03-17T23:45:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: set fsid to 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:45:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-17T23:45:35.499 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:45:35.499 INFO:teuthology.orchestra.run.smithi012.stdout:Creating mon... 2024-03-17T23:45:43.512 INFO:teuthology.orchestra.run.smithi012.stdout:create mon.a on 2024-03-17T23:45:43.903 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-17T23:45:44.144 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-17T23:45:44.449 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715.target. 2024-03-17T23:45:44.449 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715.target. 2024-03-17T23:45:44.853 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.a 2024-03-17T23:45:44.853 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-17T23:45:45.160 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715.target.wants/ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@.service. 2024-03-17T23:45:45.823 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:45 smithi012 podman[31152]: 2024-03-17 23:45:45.810909875 +0000 UTC m=+0.374208540 container init 892ac9b8fdb039357adef8196db1a297593b8403c0db5fd7f1b8765d5d2efb3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-17T23:45:45.824 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:45 smithi012 podman[31152]: 2024-03-17 23:45:45.821531872 +0000 UTC m=+0.384830477 container start 892ac9b8fdb039357adef8196db1a297593b8403c0db5fd7f1b8765d5d2efb3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-17T23:45:45.938 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-17T23:45:45.938 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-03-17T23:45:45.955 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-17T23:45:45.955 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-17T23:45:45.956 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-03-17T23:45:45.956 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to enable service . firewalld.service is not available 2024-03-17T23:45:45.957 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mon to start... 2024-03-17T23:45:45.958 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mon... 2024-03-17T23:45:46.113 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:45 smithi012 ceph-mon[31170]: mkfs 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:45:46.114 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:45 smithi012 ceph-mon[31170]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-17T23:45:46.114 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:45 smithi012 bash[31152]: 892ac9b8fdb039357adef8196db1a297593b8403c0db5fd7f1b8765d5d2efb3e 2024-03-17T23:45:46.114 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:45 smithi012 systemd[1]: Started Ceph mon.a for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:45:47.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:46 smithi012 ceph-mon[31170]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-17T23:45:47.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:46 smithi012 ceph-mon[31170]: monmap epoch 1 2024-03-17T23:45:47.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:46 smithi012 ceph-mon[31170]: fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:45:47.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:46 smithi012 ceph-mon[31170]: last_changed 2024-03-17T23:45:34.840429+0000 2024-03-17T23:45:47.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:46 smithi012 ceph-mon[31170]: created 2024-03-17T23:45:34.840429+0000 2024-03-17T23:45:47.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:46 smithi012 ceph-mon[31170]: min_mon_release 19 (squid) 2024-03-17T23:45:47.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:46 smithi012 ceph-mon[31170]: election_strategy: 1 2024-03-17T23:45:47.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:46 smithi012 ceph-mon[31170]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-03-17T23:45:47.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:46 smithi012 ceph-mon[31170]: fsmap 2024-03-17T23:45:47.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:46 smithi012 ceph-mon[31170]: osdmap e1: 0 total, 0 up, 0 in 2024-03-17T23:45:47.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:46 smithi012 ceph-mon[31170]: mgrmap e1: no daemons active 2024-03-17T23:45:47.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:46 smithi012 ceph-mon[31170]: from='client.? 172.21.15.12:0/3146186567' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-17T23:45:47.284 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout cluster: 2024-03-17T23:45:47.286 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout id: 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:45:47.286 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-17T23:45:47.286 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-17T23:45:47.286 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout services: 2024-03-17T23:45:47.287 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.745974s) 2024-03-17T23:45:47.287 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-17T23:45:47.287 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-17T23:45:47.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-17T23:45:47.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout data: 2024-03-17T23:45:47.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-17T23:45:47.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-17T23:45:47.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-17T23:45:47.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout pgs: 2024-03-17T23:45:47.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-17T23:45:47.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.548+0000 7f32f1a00700 1 Processor -- start 2024-03-17T23:45:47.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.548+0000 7f32f1a00700 1 -- start start 2024-03-17T23:45:47.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.548+0000 7f32f1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 0x7f32ec10a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:45:47.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.548+0000 7f32f1a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f32ec10a7b0 con 0x7f32ec109e50 2024-03-17T23:45:47.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.549+0000 7f32eaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 0x7f32ec10a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:45:47.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.550+0000 7f32eaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 0x7f32ec10a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35176/0 (socket says 172.21.15.12:35176) 2024-03-17T23:45:47.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.550+0000 7f32eaa00700 1 -- 172.21.15.12:0/2598667163 learned_addr learned my addr 172.21.15.12:0/2598667163 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:45:47.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.550+0000 7f32eaa00700 1 -- 172.21.15.12:0/2598667163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ec10b040 con 0x7f32ec109e50 2024-03-17T23:45:47.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.551+0000 7f32eaa00700 1 --2- 172.21.15.12:0/2598667163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 0x7f32ec10a270 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f32dc0098a0 tx=0x7f32dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9f149121858958 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:45:47.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.551+0000 7f32e9600700 1 -- 172.21.15.12:0/2598667163 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32dc004160 con 0x7f32ec109e50 2024-03-17T23:45:47.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.551+0000 7f32e9600700 1 -- 172.21.15.12:0/2598667163 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32dc004300 con 0x7f32ec109e50 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.551+0000 7f32e9600700 1 -- 172.21.15.12:0/2598667163 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32dc0036a0 con 0x7f32ec109e50 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.552+0000 7f32f1a00700 1 -- 172.21.15.12:0/2598667163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 msgr2=0x7f32ec10a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.552+0000 7f32f1a00700 1 --2- 172.21.15.12:0/2598667163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 0x7f32ec10a270 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f32dc0098a0 tx=0x7f32dc009bb0 comp rx=0 tx=0).stop 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.552+0000 7f32f1a00700 1 -- 172.21.15.12:0/2598667163 shutdown_connections 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.552+0000 7f32f1a00700 1 --2- 172.21.15.12:0/2598667163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 0x7f32ec10a270 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.552+0000 7f32f1a00700 1 -- 172.21.15.12:0/2598667163 >> 172.21.15.12:0/2598667163 conn(0x7f32ec105580 msgr2=0x7f32ec107970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.552+0000 7f32f1a00700 1 -- 172.21.15.12:0/2598667163 shutdown_connections 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.552+0000 7f32f1a00700 1 -- 172.21.15.12:0/2598667163 wait complete. 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.553+0000 7f32f1a00700 1 Processor -- start 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.553+0000 7f32f1a00700 1 -- start start 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.553+0000 7f32f1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 0x7f32ec19d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.553+0000 7f32f1a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f32ec10b510 con 0x7f32ec109e50 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.554+0000 7f32eaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 0x7f32ec19d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:45:47.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.554+0000 7f32eaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 0x7f32ec19d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35188/0 (socket says 172.21.15.12:35188) 2024-03-17T23:45:47.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.554+0000 7f32eaa00700 1 -- 172.21.15.12:0/3146186567 learned_addr learned my addr 172.21.15.12:0/3146186567 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:45:47.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.554+0000 7f32eaa00700 1 -- 172.21.15.12:0/3146186567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ec19ddd0 con 0x7f32ec109e50 2024-03-17T23:45:47.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.554+0000 7f32eaa00700 1 --2- 172.21.15.12:0/3146186567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 0x7f32ec19d890 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f32dc000c00 tx=0x7f32dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:45:47.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.555+0000 7f32e3400700 1 -- 172.21.15.12:0/3146186567 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32dc003db0 con 0x7f32ec109e50 2024-03-17T23:45:47.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.555+0000 7f32e3400700 1 -- 172.21.15.12:0/3146186567 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32dc011630 con 0x7f32ec109e50 2024-03-17T23:45:47.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.555+0000 7f32e3400700 1 -- 172.21.15.12:0/3146186567 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32dc011950 con 0x7f32ec109e50 2024-03-17T23:45:47.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.555+0000 7f32f1a00700 1 -- 172.21.15.12:0/3146186567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32ec19e060 con 0x7f32ec109e50 2024-03-17T23:45:47.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.555+0000 7f32f1a00700 1 -- 172.21.15.12:0/3146186567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32ec1a0d50 con 0x7f32ec109e50 2024-03-17T23:45:47.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.555+0000 7f32e3400700 1 -- 172.21.15.12:0/3146186567 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f32dc011af0 con 0x7f32ec109e50 2024-03-17T23:45:47.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.555+0000 7f32e3400700 1 -- 172.21.15.12:0/3146186567 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f32dc01c450 con 0x7f32ec109e50 2024-03-17T23:45:47.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.556+0000 7f32f1a00700 1 -- 172.21.15.12:0/3146186567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32ec063900 con 0x7f32ec109e50 2024-03-17T23:45:47.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.560+0000 7f32e3400700 1 -- 172.21.15.12:0/3146186567 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f32dc032020 con 0x7f32ec109e50 2024-03-17T23:45:47.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.641+0000 7f32f1a00700 1 -- 172.21.15.12:0/3146186567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f32ec02db70 con 0x7f32ec109e50 2024-03-17T23:45:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.642+0000 7f32e3400700 1 -- 172.21.15.12:0/3146186567 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f32dc004260 con 0x7f32ec109e50 2024-03-17T23:45:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.643+0000 7f32f1a00700 1 -- 172.21.15.12:0/3146186567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 msgr2=0x7f32ec19d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:45:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.643+0000 7f32f1a00700 1 --2- 172.21.15.12:0/3146186567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 0x7f32ec19d890 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f32dc000c00 tx=0x7f32dc004750 comp rx=0 tx=0).stop 2024-03-17T23:45:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.644+0000 7f32f1a00700 1 -- 172.21.15.12:0/3146186567 shutdown_connections 2024-03-17T23:45:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.644+0000 7f32f1a00700 1 --2- 172.21.15.12:0/3146186567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32ec109e50 0x7f32ec19d890 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:45:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.644+0000 7f32f1a00700 1 -- 172.21.15.12:0/3146186567 >> 172.21.15.12:0/3146186567 conn(0x7f32ec105580 msgr2=0x7f32ec194190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:45:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.644+0000 7f32f1a00700 1 -- 172.21.15.12:0/3146186567 shutdown_connections 2024-03-17T23:45:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:46.644+0000 7f32f1a00700 1 -- 172.21.15.12:0/3146186567 wait complete. 2024-03-17T23:45:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:mon is available 2024-03-17T23:45:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:Assimilating anything we can from ceph.conf... 2024-03-17T23:45:48.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:48 smithi012 ceph-mon[31170]: from='client.? 172.21.15.12:0/1846509141' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-17T23:45:48.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:48 smithi012 ceph-mon[31170]: from='client.? 172.21.15.12:0/1846509141' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-17T23:45:48.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-17T23:45:48.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [global] 2024-03-17T23:45:48.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout fsid = 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:45:48.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.12:3300,v1:172.21.15.12:6789] 2024-03-17T23:45:48.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-17T23:45:48.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-17T23:45:48.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-17T23:45:48.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-17T23:45:48.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-17T23:45:48.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-17T23:45:48.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-17T23:45:48.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-17T23:45:48.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [osd] 2024-03-17T23:45:48.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-17T23:45:48.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-17T23:45:48.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-17T23:45:48.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-17T23:45:48.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-17T23:45:48.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.884+0000 7f6b29400700 1 Processor -- start 2024-03-17T23:45:48.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.884+0000 7f6b29400700 1 -- start start 2024-03-17T23:45:48.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.885+0000 7f6b29400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 0x7f6b2407b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:45:48.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.885+0000 7f6b29400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2407b7c0 con 0x7f6b2407cd90 2024-03-17T23:45:48.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.885+0000 7f6b22800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 0x7f6b2407b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:45:48.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.885+0000 7f6b22800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 0x7f6b2407b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35190/0 (socket says 172.21.15.12:35190) 2024-03-17T23:45:48.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.885+0000 7f6b22800700 1 -- 172.21.15.12:0/3990435633 learned_addr learned my addr 172.21.15.12:0/3990435633 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:45:48.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.886+0000 7f6b22800700 1 -- 172.21.15.12:0/3990435633 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2407b940 con 0x7f6b2407cd90 2024-03-17T23:45:48.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.886+0000 7f6b22800700 1 --2- 172.21.15.12:0/3990435633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 0x7f6b2407b1f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6b140098a0 tx=0x7f6b14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a77be71ed13ccf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:45:48.913 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.887+0000 7f6b21400700 1 -- 172.21.15.12:0/3990435633 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b14004160 con 0x7f6b2407cd90 2024-03-17T23:45:48.913 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.887+0000 7f6b21400700 1 -- 172.21.15.12:0/3990435633 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b14004300 con 0x7f6b2407cd90 2024-03-17T23:45:48.913 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.887+0000 7f6b21400700 1 -- 172.21.15.12:0/3990435633 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b140036a0 con 0x7f6b2407cd90 2024-03-17T23:45:48.913 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.887+0000 7f6b29400700 1 -- 172.21.15.12:0/3990435633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 msgr2=0x7f6b2407b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:45:48.913 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.887+0000 7f6b29400700 1 --2- 172.21.15.12:0/3990435633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 0x7f6b2407b1f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6b140098a0 tx=0x7f6b14009bb0 comp rx=0 tx=0).stop 2024-03-17T23:45:48.913 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.887+0000 7f6b29400700 1 -- 172.21.15.12:0/3990435633 shutdown_connections 2024-03-17T23:45:48.913 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.887+0000 7f6b29400700 1 --2- 172.21.15.12:0/3990435633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 0x7f6b2407b1f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:45:48.913 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.888+0000 7f6b29400700 1 -- 172.21.15.12:0/3990435633 >> 172.21.15.12:0/3990435633 conn(0x7f6b24103240 msgr2=0x7f6b24105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:45:48.913 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.888+0000 7f6b29400700 1 -- 172.21.15.12:0/3990435633 shutdown_connections 2024-03-17T23:45:48.913 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.888+0000 7f6b29400700 1 -- 172.21.15.12:0/3990435633 wait complete. 2024-03-17T23:45:48.913 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.889+0000 7f6b29400700 1 Processor -- start 2024-03-17T23:45:48.913 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.890+0000 7f6b29400700 1 -- start start 2024-03-17T23:45:48.913 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.890+0000 7f6b29400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 0x7f6b2410af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.890+0000 7f6b29400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2407be70 con 0x7f6b2407cd90 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.890+0000 7f6b22800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 0x7f6b2410af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.890+0000 7f6b22800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 0x7f6b2410af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35196/0 (socket says 172.21.15.12:35196) 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.890+0000 7f6b22800700 1 -- 172.21.15.12:0/1846509141 learned_addr learned my addr 172.21.15.12:0/1846509141 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.891+0000 7f6b22800700 1 -- 172.21.15.12:0/1846509141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2410b450 con 0x7f6b2407cd90 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.891+0000 7f6b22800700 1 --2- 172.21.15.12:0/1846509141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 0x7f6b2410af10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6b14000c00 tx=0x7f6b14004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.891+0000 7f6b1b400700 1 -- 172.21.15.12:0/1846509141 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b14003db0 con 0x7f6b2407cd90 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.892+0000 7f6b1b400700 1 -- 172.21.15.12:0/1846509141 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b14011630 con 0x7f6b2407cd90 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.892+0000 7f6b29400700 1 -- 172.21.15.12:0/1846509141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b2410b6e0 con 0x7f6b2407cd90 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.892+0000 7f6b1b400700 1 -- 172.21.15.12:0/1846509141 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b14011950 con 0x7f6b2407cd90 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.892+0000 7f6b29400700 1 -- 172.21.15.12:0/1846509141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b24107ca0 con 0x7f6b2407cd90 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.892+0000 7f6b1b400700 1 -- 172.21.15.12:0/1846509141 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6b14011af0 con 0x7f6b2407cd90 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.892+0000 7f6b1b400700 1 -- 172.21.15.12:0/1846509141 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6b1401c450 con 0x7f6b2407cd90 2024-03-17T23:45:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.893+0000 7f6b29400700 1 -- 172.21.15.12:0/1846509141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b2410b8b0 con 0x7f6b2407cd90 2024-03-17T23:45:48.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.896+0000 7f6b1b400700 1 -- 172.21.15.12:0/1846509141 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6b2410b8b0 con 0x7f6b2407cd90 2024-03-17T23:45:48.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.983+0000 7f6b29400700 1 -- 172.21.15.12:0/1846509141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f6b24063930 con 0x7f6b2407cd90 2024-03-17T23:45:48.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.991+0000 7f6b1b400700 1 -- 172.21.15.12:0/1846509141 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f6b24063930 con 0x7f6b2407cd90 2024-03-17T23:45:48.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.991+0000 7f6b1b400700 1 -- 172.21.15.12:0/1846509141 <== mon.0 v2:172.21.15.12:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6b14004260 con 0x7f6b2407cd90 2024-03-17T23:45:48.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.993+0000 7f6b29400700 1 -- 172.21.15.12:0/1846509141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 msgr2=0x7f6b2410af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:45:48.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.993+0000 7f6b29400700 1 --2- 172.21.15.12:0/1846509141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 0x7f6b2410af10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6b14000c00 tx=0x7f6b14004750 comp rx=0 tx=0).stop 2024-03-17T23:45:48.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.993+0000 7f6b29400700 1 -- 172.21.15.12:0/1846509141 shutdown_connections 2024-03-17T23:45:48.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.993+0000 7f6b29400700 1 --2- 172.21.15.12:0/1846509141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b2407cd90 0x7f6b2410af10 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:45:48.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.993+0000 7f6b29400700 1 -- 172.21.15.12:0/1846509141 >> 172.21.15.12:0/1846509141 conn(0x7f6b24103240 msgr2=0x7f6b241055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:45:48.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.994+0000 7f6b29400700 1 -- 172.21.15.12:0/1846509141 shutdown_connections 2024-03-17T23:45:48.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:47.994+0000 7f6b29400700 1 -- 172.21.15.12:0/1846509141 wait complete. 2024-03-17T23:45:48.915 INFO:teuthology.orchestra.run.smithi012.stdout:Generating new minimal ceph.conf... 2024-03-17T23:45:49.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:49 smithi012 ceph-mon[31170]: from='client.? 172.21.15.12:0/3081071066' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:45:50.241 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.495+0000 7f721e800700 1 Processor -- start 2024-03-17T23:45:50.242 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.495+0000 7f721e800700 1 -- start start 2024-03-17T23:45:50.242 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.495+0000 7f721e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 0x7f721810aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:45:50.242 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.496+0000 7f721e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7218075a40 con 0x7f72181086c0 2024-03-17T23:45:50.242 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.496+0000 7f7217400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 0x7f721810aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:45:50.242 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.496+0000 7f7217400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 0x7f721810aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35208/0 (socket says 172.21.15.12:35208) 2024-03-17T23:45:50.242 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.496+0000 7f7217400700 1 -- 172.21.15.12:0/3112697647 learned_addr learned my addr 172.21.15.12:0/3112697647 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:45:50.242 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.497+0000 7f7217400700 1 -- 172.21.15.12:0/3112697647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f721810aff0 con 0x7f72181086c0 2024-03-17T23:45:50.242 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.497+0000 7f7217400700 1 --2- 172.21.15.12:0/3112697647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 0x7f721810aab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f72080098a0 tx=0x7f7208009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97db319b0fc10f1d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:45:50.242 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.497+0000 7f7216000700 1 -- 172.21.15.12:0/3112697647 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7208004160 con 0x7f72181086c0 2024-03-17T23:45:50.242 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.497+0000 7f7216000700 1 -- 172.21.15.12:0/3112697647 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7208004300 con 0x7f72181086c0 2024-03-17T23:45:50.243 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.497+0000 7f7216000700 1 -- 172.21.15.12:0/3112697647 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7208011b50 con 0x7f72181086c0 2024-03-17T23:45:50.243 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.498+0000 7f721e800700 1 -- 172.21.15.12:0/3112697647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 msgr2=0x7f721810aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:45:50.243 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.498+0000 7f721e800700 1 --2- 172.21.15.12:0/3112697647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 0x7f721810aab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f72080098a0 tx=0x7f7208009bb0 comp rx=0 tx=0).stop 2024-03-17T23:45:50.243 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.498+0000 7f721e800700 1 -- 172.21.15.12:0/3112697647 shutdown_connections 2024-03-17T23:45:50.243 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.498+0000 7f721e800700 1 --2- 172.21.15.12:0/3112697647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 0x7f721810aab0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:45:50.243 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.498+0000 7f721e800700 1 -- 172.21.15.12:0/3112697647 >> 172.21.15.12:0/3112697647 conn(0x7f7218102130 msgr2=0x7f7218104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:45:50.243 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.498+0000 7f721e800700 1 -- 172.21.15.12:0/3112697647 shutdown_connections 2024-03-17T23:45:50.243 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.498+0000 7f721e800700 1 -- 172.21.15.12:0/3112697647 wait complete. 2024-03-17T23:45:50.243 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.499+0000 7f721e800700 1 Processor -- start 2024-03-17T23:45:50.243 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.499+0000 7f721e800700 1 -- start start 2024-03-17T23:45:50.243 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.500+0000 7f721e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 0x7f721819ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:45:50.243 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.500+0000 7f721e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f721810c020 con 0x7f72181086c0 2024-03-17T23:45:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.500+0000 7f7217400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 0x7f721819ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:45:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.500+0000 7f7217400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 0x7f721819ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35222/0 (socket says 172.21.15.12:35222) 2024-03-17T23:45:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.500+0000 7f7217400700 1 -- 172.21.15.12:0/3081071066 learned_addr learned my addr 172.21.15.12:0/3081071066 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:45:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.500+0000 7f7217400700 1 -- 172.21.15.12:0/3081071066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f721819e300 con 0x7f72181086c0 2024-03-17T23:45:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.501+0000 7f7217400700 1 --2- 172.21.15.12:0/3081071066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 0x7f721819ddc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7208011b60 tx=0x7f7208011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:45:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.501+0000 7f720fe00700 1 -- 172.21.15.12:0/3081071066 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7208003b20 con 0x7f72181086c0 2024-03-17T23:45:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.501+0000 7f721e800700 1 -- 172.21.15.12:0/3081071066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f721819e590 con 0x7f72181086c0 2024-03-17T23:45:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.501+0000 7f720fe00700 1 -- 172.21.15.12:0/3081071066 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7208003cc0 con 0x7f72181086c0 2024-03-17T23:45:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.501+0000 7f720fe00700 1 -- 172.21.15.12:0/3081071066 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7208024840 con 0x7f72181086c0 2024-03-17T23:45:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.501+0000 7f721e800700 1 -- 172.21.15.12:0/3081071066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72181a1280 con 0x7f72181086c0 2024-03-17T23:45:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.502+0000 7f720fe00700 1 -- 172.21.15.12:0/3081071066 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7208022020 con 0x7f72181086c0 2024-03-17T23:45:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.502+0000 7f720fe00700 1 -- 172.21.15.12:0/3081071066 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7208011e00 con 0x7f72181086c0 2024-03-17T23:45:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.503+0000 7f721e800700 1 -- 172.21.15.12:0/3081071066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7218063900 con 0x7f72181086c0 2024-03-17T23:45:50.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.506+0000 7f720fe00700 1 -- 172.21.15.12:0/3081071066 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7208043dd0 con 0x7f72181086c0 2024-03-17T23:45:50.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.585+0000 7f721e800700 1 -- 172.21.15.12:0/3081071066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f72181a1960 con 0x7f72181086c0 2024-03-17T23:45:50.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.586+0000 7f720fe00700 1 -- 172.21.15.12:0/3081071066 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f7208032020 con 0x7f72181086c0 2024-03-17T23:45:50.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.587+0000 7f721e800700 1 -- 172.21.15.12:0/3081071066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 msgr2=0x7f721819ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:45:50.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.587+0000 7f721e800700 1 --2- 172.21.15.12:0/3081071066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 0x7f721819ddc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7208011b60 tx=0x7f7208011c40 comp rx=0 tx=0).stop 2024-03-17T23:45:50.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.587+0000 7f721e800700 1 -- 172.21.15.12:0/3081071066 shutdown_connections 2024-03-17T23:45:50.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.587+0000 7f721e800700 1 --2- 172.21.15.12:0/3081071066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72181086c0 0x7f721819ddc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:45:50.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.587+0000 7f721e800700 1 -- 172.21.15.12:0/3081071066 >> 172.21.15.12:0/3081071066 conn(0x7f7218102130 msgr2=0x7f7218102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:45:50.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.588+0000 7f721e800700 1 -- 172.21.15.12:0/3081071066 shutdown_connections 2024-03-17T23:45:50.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:49.588+0000 7f721e800700 1 -- 172.21.15.12:0/3081071066 wait complete. 2024-03-17T23:45:50.245 INFO:teuthology.orchestra.run.smithi012.stdout:Restarting the monitor... 2024-03-17T23:45:50.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:50 smithi012 systemd[1]: Stopping Ceph mon.a for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-17T23:45:51.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:50 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a[31166]: 2024-03-17T23:45:50.749+0000 7f6c48a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-17T23:45:51.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:50 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a[31166]: 2024-03-17T23:45:50.749+0000 7f6c48a00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-17T23:45:51.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:50 smithi012 podman[31415]: 2024-03-17 23:45:50.801759769 +0000 UTC m=+0.189332598 container died 892ac9b8fdb039357adef8196db1a297593b8403c0db5fd7f1b8765d5d2efb3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-17T23:45:51.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:51 smithi012 podman[31415]: 2024-03-17 23:45:51.383656365 +0000 UTC m=+0.771229191 container cleanup 892ac9b8fdb039357adef8196db1a297593b8403c0db5fd7f1b8765d5d2efb3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-17T23:45:51.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:51 smithi012 bash[31415]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a 2024-03-17T23:45:51.834 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:51 smithi012 podman[31429]: 2024-03-17 23:45:51.559474065 +0000 UTC m=+0.745438396 container remove 892ac9b8fdb039357adef8196db1a297593b8403c0db5fd7f1b8765d5d2efb3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-17T23:45:52.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 systemd[1]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-17T23:45:52.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 systemd[1]: Stopped Ceph mon.a for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:45:52.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 systemd[1]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.a.service: Consumed 1.998s CPU time. 2024-03-17T23:45:52.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 systemd[1]: Starting Ceph mon.a for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-17T23:45:52.783 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 podman[31522]: 2024-03-17 23:45:52.39675632 +0000 UTC m=+0.039219141 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:45:52.783 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 podman[31522]: 2024-03-17 23:45:52.502203732 +0000 UTC m=+0.144666583 container create ca282921d1340c62ce4d0fd52fdea1dd5551421e4e7f3b2cb4d1f746bfdd9d49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-17T23:45:52.783 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 podman[31522]: 2024-03-17 23:45:52.781926283 +0000 UTC m=+0.424389196 container init ca282921d1340c62ce4d0fd52fdea1dd5551421e4e7f3b2cb4d1f746bfdd9d49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-17T23:45:52.873 INFO:teuthology.orchestra.run.smithi012.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-17T23:45:53.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 podman[31522]: 2024-03-17 23:45:52.792812691 +0000 UTC m=+0.435275531 container start ca282921d1340c62ce4d0fd52fdea1dd5551421e4e7f3b2cb4d1f746bfdd9d49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307) 2024-03-17T23:45:53.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: set uid:gid to 167:167 (ceph:ceph) 2024-03-17T23:45:53.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-17T23:45:53.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: pidfile_write: ignore empty --pid-file 2024-03-17T23:45:53.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: load: jerasure load: lrc 2024-03-17T23:45:53.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: RocksDB version: 7.9.2 2024-03-17T23:45:53.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Git sha 0 2024-03-17T23:45:53.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-17T23:45:53.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: DB SUMMARY 2024-03-17T23:45:53.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: DB Session ID: 98TNVGGLDWOEJULLVKPL 2024-03-17T23:45:53.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: CURRENT file: CURRENT 2024-03-17T23:45:53.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: IDENTITY file: IDENTITY 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104924 ; 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.error_if_exists: 0 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.create_if_missing: 0 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.paranoid_checks: 1 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.env: 0x5647640acc60 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.fs: PosixFileSystem 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.info_log: 0x564765835b00 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_file_opening_threads: 16 2024-03-17T23:45:53.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.statistics: (nil) 2024-03-17T23:45:53.172 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.use_fsync: 0 2024-03-17T23:45:53.172 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_log_file_size: 0 2024-03-17T23:45:53.172 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-17T23:45:53.172 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-17T23:45:53.172 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.keep_log_file_num: 1000 2024-03-17T23:45:53.172 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.recycle_log_file_num: 0 2024-03-17T23:45:53.172 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.allow_fallocate: 1 2024-03-17T23:45:53.173 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.allow_mmap_reads: 0 2024-03-17T23:45:53.173 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.allow_mmap_writes: 0 2024-03-17T23:45:53.173 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.use_direct_reads: 0 2024-03-17T23:45:53.173 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-17T23:45:53.173 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.create_missing_column_families: 0 2024-03-17T23:45:53.173 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.db_log_dir: 2024-03-17T23:45:53.173 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.wal_dir: 2024-03-17T23:45:53.173 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-17T23:45:53.173 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-17T23:45:53.173 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-17T23:45:53.173 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-17T23:45:53.173 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-17T23:45:53.174 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-17T23:45:53.174 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.advise_random_on_open: 1 2024-03-17T23:45:53.174 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.db_write_buffer_size: 0 2024-03-17T23:45:53.174 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.write_buffer_manager: 0x5647658081e0 2024-03-17T23:45:53.174 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-17T23:45:53.174 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-17T23:45:53.174 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-17T23:45:53.174 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.rate_limiter: (nil) 2024-03-17T23:45:53.174 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-17T23:45:53.174 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.wal_recovery_mode: 2 2024-03-17T23:45:53.174 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.enable_thread_tracking: 0 2024-03-17T23:45:53.174 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.enable_pipelined_write: 0 2024-03-17T23:45:53.174 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.unordered_write: 0 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.row_cache: None 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.wal_filter: None 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.allow_ingest_behind: 0 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.two_write_queues: 0 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.manual_wal_flush: 0 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.wal_compression: 0 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.atomic_flush: 0 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-17T23:45:53.175 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.log_readahead_size: 0 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.best_efforts_recovery: 0 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.allow_data_in_errors: 0 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.db_host_id: __hostname__ 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_background_jobs: 2 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_background_compactions: -1 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_subcompactions: 1 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-17T23:45:53.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-17T23:45:53.177 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-17T23:45:53.177 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_total_wal_size: 0 2024-03-17T23:45:53.177 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-17T23:45:53.177 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-17T23:45:53.177 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-17T23:45:53.177 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-17T23:45:53.177 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_open_files: -1 2024-03-17T23:45:53.178 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.bytes_per_sync: 0 2024-03-17T23:45:53.178 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-17T23:45:53.178 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-17T23:45:53.178 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compaction_readahead_size: 0 2024-03-17T23:45:53.178 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_background_flushes: -1 2024-03-17T23:45:53.178 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Compression algorithms supported: 2024-03-17T23:45:53.179 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-17T23:45:53.179 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: kZSTD supported: 0 2024-03-17T23:45:53.179 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: kXpressCompression supported: 0 2024-03-17T23:45:53.179 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: kLZ4HCCompression supported: 1 2024-03-17T23:45:53.179 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: kZlibCompression supported: 1 2024-03-17T23:45:53.179 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: kSnappyCompression supported: 1 2024-03-17T23:45:53.179 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: kLZ4Compression supported: 1 2024-03-17T23:45:53.180 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: kBZip2Compression supported: 0 2024-03-17T23:45:53.180 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-17T23:45:53.180 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-17T23:45:53.180 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-17T23:45:53.180 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-17T23:45:53.180 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-17T23:45:53.181 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.merge_operator: 2024-03-17T23:45:53.181 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compaction_filter: None 2024-03-17T23:45:53.181 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compaction_filter_factory: None 2024-03-17T23:45:53.181 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.sst_partitioner_factory: None 2024-03-17T23:45:53.181 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-17T23:45:53.181 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-17T23:45:53.181 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564765835c20) 2024-03-17T23:45:53.182 INFO:journalctl@ceph.mon.a.smithi012.stdout: cache_index_and_filter_blocks: 1 2024-03-17T23:45:53.182 INFO:journalctl@ceph.mon.a.smithi012.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-17T23:45:53.182 INFO:journalctl@ceph.mon.a.smithi012.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-17T23:45:53.182 INFO:journalctl@ceph.mon.a.smithi012.stdout: pin_top_level_index_and_filter: 1 2024-03-17T23:45:53.182 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_type: 0 2024-03-17T23:45:53.182 INFO:journalctl@ceph.mon.a.smithi012.stdout: data_block_index_type: 0 2024-03-17T23:45:53.183 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_shortening: 1 2024-03-17T23:45:53.183 INFO:journalctl@ceph.mon.a.smithi012.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-17T23:45:53.183 INFO:journalctl@ceph.mon.a.smithi012.stdout: checksum: 4 2024-03-17T23:45:53.183 INFO:journalctl@ceph.mon.a.smithi012.stdout: no_block_cache: 0 2024-03-17T23:45:53.183 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache: 0x56476583b090 2024-03-17T23:45:53.183 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_name: BinnedLRUCache 2024-03-17T23:45:53.184 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_options: 2024-03-17T23:45:53.184 INFO:journalctl@ceph.mon.a.smithi012.stdout: capacity : 536870912 2024-03-17T23:45:53.184 INFO:journalctl@ceph.mon.a.smithi012.stdout: num_shard_bits : 4 2024-03-17T23:45:53.184 INFO:journalctl@ceph.mon.a.smithi012.stdout: strict_capacity_limit : 0 2024-03-17T23:45:53.184 INFO:journalctl@ceph.mon.a.smithi012.stdout: high_pri_pool_ratio: 0.000 2024-03-17T23:45:53.184 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_compressed: (nil) 2024-03-17T23:45:53.185 INFO:journalctl@ceph.mon.a.smithi012.stdout: persistent_cache: (nil) 2024-03-17T23:45:53.185 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_size: 4096 2024-03-17T23:45:53.185 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_size_deviation: 10 2024-03-17T23:45:53.185 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_restart_interval: 16 2024-03-17T23:45:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_block_restart_interval: 1 2024-03-17T23:45:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout: metadata_block_size: 4096 2024-03-17T23:45:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout: partition_filters: 0 2024-03-17T23:45:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout: use_delta_encoding: 1 2024-03-17T23:45:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout: filter_policy: bloomfilter 2024-03-17T23:45:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout: whole_key_filtering: 1 2024-03-17T23:45:53.187 INFO:journalctl@ceph.mon.a.smithi012.stdout: verify_compression: 0 2024-03-17T23:45:53.187 INFO:journalctl@ceph.mon.a.smithi012.stdout: read_amp_bytes_per_bit: 0 2024-03-17T23:45:53.187 INFO:journalctl@ceph.mon.a.smithi012.stdout: format_version: 5 2024-03-17T23:45:53.187 INFO:journalctl@ceph.mon.a.smithi012.stdout: enable_index_compression: 1 2024-03-17T23:45:53.187 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_align: 0 2024-03-17T23:45:53.187 INFO:journalctl@ceph.mon.a.smithi012.stdout: max_auto_readahead_size: 262144 2024-03-17T23:45:53.187 INFO:journalctl@ceph.mon.a.smithi012.stdout: prepopulate_block_cache: 0 2024-03-17T23:45:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout: initial_auto_readahead_size: 8192 2024-03-17T23:45:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout: num_file_reads_for_auto_readahead: 2 2024-03-17T23:45:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.write_buffer_size: 33554432 2024-03-17T23:45:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_write_buffer_number: 2 2024-03-17T23:45:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compression: NoCompression 2024-03-17T23:45:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.bottommost_compression: Disabled 2024-03-17T23:45:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.prefix_extractor: nullptr 2024-03-17T23:45:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-17T23:45:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.num_levels: 7 2024-03-17T23:45:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-17T23:45:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-17T23:45:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-17T23:45:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-17T23:45:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-17T23:45:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-17T23:45:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-17T23:45:53.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-17T23:45:53.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-17T23:45:53.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-17T23:45:53.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-17T23:45:53.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-17T23:45:53.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-17T23:45:53.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compression_opts.level: 32767 2024-03-17T23:45:53.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compression_opts.strategy: 0 2024-03-17T23:45:53.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-17T23:45:53.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-17T23:45:53.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-17T23:45:53.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-17T23:45:53.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compression_opts.enabled: false 2024-03-17T23:45:53.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-17T23:45:53.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-17T23:45:53.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-17T23:45:53.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-17T23:45:53.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.target_file_size_base: 67108864 2024-03-17T23:45:53.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-17T23:45:53.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-17T23:45:53.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-17T23:45:53.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-17T23:45:53.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-17T23:45:53.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-17T23:45:53.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-17T23:45:53.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-17T23:45:53.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-17T23:45:53.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-17T23:45:53.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-17T23:45:53.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-17T23:45:53.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-17T23:45:53.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-17T23:45:53.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.arena_block_size: 1048576 2024-03-17T23:45:53.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-17T23:45:53.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-17T23:45:53.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.disable_auto_compactions: 0 2024-03-17T23:45:53.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-17T23:45:53.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-17T23:45:53.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-17T23:45:53.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-17T23:45:53.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-17T23:45:53.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-17T23:45:53.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-17T23:45:53.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-17T23:45:53.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-17T23:45:53.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-17T23:45:53.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.table_properties_collectors: 2024-03-17T23:45:53.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.inplace_update_support: 0 2024-03-17T23:45:53.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-17T23:45:53.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-17T23:45:53.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-17T23:45:53.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-17T23:45:53.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.bloom_locality: 0 2024-03-17T23:45:53.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.max_successive_merges: 0 2024-03-17T23:45:53.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-17T23:45:53.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.paranoid_file_checks: 0 2024-03-17T23:45:53.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.force_consistency_checks: 1 2024-03-17T23:45:53.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.report_bg_io_stats: 0 2024-03-17T23:45:53.202 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.ttl: 2592000 2024-03-17T23:45:53.202 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-17T23:45:53.202 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-17T23:45:53.202 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-17T23:45:53.203 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.enable_blob_files: false 2024-03-17T23:45:53.203 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.min_blob_size: 0 2024-03-17T23:45:53.203 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.blob_file_size: 268435456 2024-03-17T23:45:53.203 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-17T23:45:53.203 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-17T23:45:53.203 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-17T23:45:53.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-17T23:45:53.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-17T23:45:53.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.blob_file_starting_level: 0 2024-03-17T23:45:53.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-17T23:45:53.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-17T23:45:53.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-17T23:45:53.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 23c809ba-380d-4ae5-a9f4-59dffe1b4b84 2024-03-17T23:45:53.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710719152853491, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-17T23:45:53.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-17T23:45:53.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710719152856187, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100073, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97985, "index_size": 306, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14108, "raw_average_key_size": 49, "raw_value_size": 90311, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710719152, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "23c809ba-380d-4ae5-a9f4-59dffe1b4b84", "db_session_id": "98TNVGGLDWOEJULLVKPL", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-17T23:45:53.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710719152856406, "job": 1, "event": "recovery_finished"} 2024-03-17T23:45:53.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-17T23:45:53.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-17T23:45:53.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56476592e000 2024-03-17T23:45:53.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: DB pointer 0x564765918000 2024-03-17T23:45:53.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-17T23:45:53.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-17T23:45:53.207 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** DB Stats ** 2024-03-17T23:45:53.207 INFO:journalctl@ceph.mon.a.smithi012.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-17T23:45:53.207 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-17T23:45:53.207 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-17T23:45:53.207 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-17T23:45:53.208 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-17T23:45:53.208 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-17T23:45:53.208 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-17T23:45:53.208 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-17T23:45:53.208 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** Compaction Stats [default] ** 2024-03-17T23:45:53.208 INFO:journalctl@ceph.mon.a.smithi012.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-17T23:45:53.209 INFO:journalctl@ceph.mon.a.smithi012.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-17T23:45:53.209 INFO:journalctl@ceph.mon.a.smithi012.stdout: L0 2/0 99.56 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-17T23:45:53.209 INFO:journalctl@ceph.mon.a.smithi012.stdout: Sum 2/0 99.56 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-17T23:45:53.209 INFO:journalctl@ceph.mon.a.smithi012.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-17T23:45:53.209 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-17T23:45:53.209 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** Compaction Stats [default] ** 2024-03-17T23:45:53.210 INFO:journalctl@ceph.mon.a.smithi012.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-17T23:45:53.210 INFO:journalctl@ceph.mon.a.smithi012.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-17T23:45:53.210 INFO:journalctl@ceph.mon.a.smithi012.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 58.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-17T23:45:53.210 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-17T23:45:53.210 INFO:journalctl@ceph.mon.a.smithi012.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-17T23:45:53.210 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-17T23:45:53.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-17T23:45:53.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-17T23:45:53.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-17T23:45:53.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-17T23:45:53.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-17T23:45:53.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-17T23:45:53.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative compaction: 0.00 GB write, 9.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-17T23:45:53.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval compaction: 0.00 GB write, 9.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-17T23:45:53.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-17T23:45:53.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: Block cache BinnedLRUCache@0x56476583b090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-17T23:45:53.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-17T23:45:53.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-17T23:45:53.213 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-17T23:45:53.213 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: starting mon.a rank 0 at public addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] at bind addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:45:53.213 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 bash[31522]: ca282921d1340c62ce4d0fd52fdea1dd5551421e4e7f3b2cb4d1f746bfdd9d49 2024-03-17T23:45:53.213 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: mon.a@-1(???) e1 preinit fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:45:53.213 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: mon.a@-1(???).mds e1 new map 2024-03-17T23:45:53.213 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: mon.a@-1(???).mds e1 print_map 2024-03-17T23:45:53.214 INFO:journalctl@ceph.mon.a.smithi012.stdout: e1 2024-03-17T23:45:53.214 INFO:journalctl@ceph.mon.a.smithi012.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-17T23:45:53.214 INFO:journalctl@ceph.mon.a.smithi012.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-17T23:45:53.214 INFO:journalctl@ceph.mon.a.smithi012.stdout: legacy client fscid: -1 2024-03-17T23:45:53.214 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-17T23:45:53.214 INFO:journalctl@ceph.mon.a.smithi012.stdout: No filesystems configured 2024-03-17T23:45:53.214 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-17T23:45:53.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-17T23:45:53.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-17T23:45:53.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-17T23:45:53.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-17T23:45:53.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 systemd[1]: Started Ceph mon.a for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:45:53.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-17T23:45:53.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: monmap epoch 1 2024-03-17T23:45:53.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:45:53.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: last_changed 2024-03-17T23:45:34.840429+0000 2024-03-17T23:45:53.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: created 2024-03-17T23:45:34.840429+0000 2024-03-17T23:45:53.216 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: min_mon_release 19 (squid) 2024-03-17T23:45:53.216 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: election_strategy: 1 2024-03-17T23:45:53.216 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-03-17T23:45:53.216 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: fsmap 2024-03-17T23:45:53.216 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: osdmap e1: 0 total, 0 up, 0 in 2024-03-17T23:45:53.216 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:52 smithi012 ceph-mon[31541]: mgrmap e1: no daemons active 2024-03-17T23:45:54.465 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.756+0000 7fbaf8a00700 1 Processor -- start 2024-03-17T23:45:54.466 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.756+0000 7fbaf8a00700 1 -- start start 2024-03-17T23:45:54.466 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.757+0000 7fbaf8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 0x7fbaf4108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:45:54.466 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.757+0000 7fbaf8a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf4108640 con 0x7fbaf4107c50 2024-03-17T23:45:54.466 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.757+0000 7fbaf1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 0x7fbaf4108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:45:54.466 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.757+0000 7fbaf1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 0x7fbaf4108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35228/0 (socket says 172.21.15.12:35228) 2024-03-17T23:45:54.467 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.757+0000 7fbaf1e00700 1 -- 172.21.15.12:0/3433570582 learned_addr learned my addr 172.21.15.12:0/3433570582 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:45:54.467 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.758+0000 7fbaf1e00700 1 -- 172.21.15.12:0/3433570582 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf4108e50 con 0x7fbaf4107c50 2024-03-17T23:45:54.467 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.758+0000 7fbaf1e00700 1 --2- 172.21.15.12:0/3433570582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 0x7fbaf4108070 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbae40098a0 tx=0x7fbae4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9c381aa1df6c59 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:45:54.467 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.758+0000 7fbaf1400700 1 -- 172.21.15.12:0/3433570582 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbae4004160 con 0x7fbaf4107c50 2024-03-17T23:45:54.467 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.759+0000 7fbaf1400700 1 -- 172.21.15.12:0/3433570582 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbae4004300 con 0x7fbaf4107c50 2024-03-17T23:45:54.470 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.759+0000 7fbaf1400700 1 -- 172.21.15.12:0/3433570582 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbae4011b50 con 0x7fbaf4107c50 2024-03-17T23:45:54.470 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.759+0000 7fbaf8a00700 1 -- 172.21.15.12:0/3433570582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 msgr2=0x7fbaf4108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.759+0000 7fbaf8a00700 1 --2- 172.21.15.12:0/3433570582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 0x7fbaf4108070 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbae40098a0 tx=0x7fbae4009bb0 comp rx=0 tx=0).stop 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.759+0000 7fbaf8a00700 1 -- 172.21.15.12:0/3433570582 shutdown_connections 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.759+0000 7fbaf8a00700 1 --2- 172.21.15.12:0/3433570582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 0x7fbaf4108070 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.759+0000 7fbaf8a00700 1 -- 172.21.15.12:0/3433570582 >> 172.21.15.12:0/3433570582 conn(0x7fbaf41031f0 msgr2=0x7fbaf4105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.759+0000 7fbaf8a00700 1 -- 172.21.15.12:0/3433570582 shutdown_connections 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.759+0000 7fbaf8a00700 1 -- 172.21.15.12:0/3433570582 wait complete. 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.760+0000 7fbaf8a00700 1 Processor -- start 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.761+0000 7fbaf8a00700 1 -- start start 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.761+0000 7fbaf8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 0x7fbaf419de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.761+0000 7fbaf8a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf410c200 con 0x7fbaf4107c50 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.761+0000 7fbaf1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 0x7fbaf419de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.761+0000 7fbaf1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 0x7fbaf419de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35236/0 (socket says 172.21.15.12:35236) 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.761+0000 7fbaf1e00700 1 -- 172.21.15.12:0/672212312 learned_addr learned my addr 172.21.15.12:0/672212312 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:45:54.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.762+0000 7fbaf1e00700 1 -- 172.21.15.12:0/672212312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf419e3b0 con 0x7fbaf4107c50 2024-03-17T23:45:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.762+0000 7fbaf1e00700 1 --2- 172.21.15.12:0/672212312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 0x7fbaf419de70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbae4000c00 tx=0x7fbae4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:45:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.762+0000 7fbaeaa00700 1 -- 172.21.15.12:0/672212312 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbae4003b20 con 0x7fbaf4107c50 2024-03-17T23:45:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.762+0000 7fbaeaa00700 1 -- 172.21.15.12:0/672212312 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbae4003cc0 con 0x7fbaf4107c50 2024-03-17T23:45:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.762+0000 7fbaf8a00700 1 -- 172.21.15.12:0/672212312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaf419e640 con 0x7fbaf4107c50 2024-03-17T23:45:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.762+0000 7fbaeaa00700 1 -- 172.21.15.12:0/672212312 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbae4024910 con 0x7fbaf4107c50 2024-03-17T23:45:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.763+0000 7fbaf8a00700 1 -- 172.21.15.12:0/672212312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaf41a1330 con 0x7fbaf4107c50 2024-03-17T23:45:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.763+0000 7fbaeaa00700 1 -- 172.21.15.12:0/672212312 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbae4022020 con 0x7fbaf4107c50 2024-03-17T23:45:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.763+0000 7fbaeaa00700 1 -- 172.21.15.12:0/672212312 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbae4011e00 con 0x7fbaf4107c50 2024-03-17T23:45:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.764+0000 7fbaf8a00700 1 -- 172.21.15.12:0/672212312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaf4063900 con 0x7fbaf4107c50 2024-03-17T23:45:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.767+0000 7fbaeaa00700 1 -- 172.21.15.12:0/672212312 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fbae402ddf0 con 0x7fbaf4107c50 2024-03-17T23:45:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.845+0000 7fbaf8a00700 1 -- 172.21.15.12:0/672212312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fbaf41a1500 con 0x7fbaf4107c50 2024-03-17T23:45:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.848+0000 7fbaeaa00700 1 -- 172.21.15.12:0/672212312 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fbae4032030 con 0x7fbaf4107c50 2024-03-17T23:45:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.849+0000 7fbaf8a00700 1 -- 172.21.15.12:0/672212312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 msgr2=0x7fbaf419de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:45:54.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.849+0000 7fbaf8a00700 1 --2- 172.21.15.12:0/672212312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 0x7fbaf419de70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbae4000c00 tx=0x7fbae4011c10 comp rx=0 tx=0).stop 2024-03-17T23:45:54.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.849+0000 7fbaf8a00700 1 -- 172.21.15.12:0/672212312 shutdown_connections 2024-03-17T23:45:54.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.849+0000 7fbaf8a00700 1 --2- 172.21.15.12:0/672212312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaf4107c50 0x7fbaf419de70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:45:54.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.849+0000 7fbaf8a00700 1 -- 172.21.15.12:0/672212312 >> 172.21.15.12:0/672212312 conn(0x7fbaf41031f0 msgr2=0x7fbaf4103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:45:54.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.849+0000 7fbaf8a00700 1 -- 172.21.15.12:0/672212312 shutdown_connections 2024-03-17T23:45:54.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:53.850+0000 7fbaf8a00700 1 -- 172.21.15.12:0/672212312 wait complete. 2024-03-17T23:45:54.473 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-17T23:45:54.494 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-17T23:45:54.495 INFO:teuthology.orchestra.run.smithi012.stdout:Creating mgr... 2024-03-17T23:45:54.496 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-17T23:45:54.497 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-17T23:45:54.950 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mgr.x 2024-03-17T23:45:54.951 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-17T23:45:55.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:54 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/672212312' entity='client.admin' 2024-03-17T23:45:55.208 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715.target.wants/ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@.service. 2024-03-17T23:45:55.593 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:55 smithi012 systemd[1]: Starting Ceph mgr.x for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-17T23:45:55.859 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:55 smithi012 podman[31756]: 2024-03-17 23:45:55.502917699 +0000 UTC m=+0.039209356 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:45:55.859 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:55 smithi012 podman[31756]: 2024-03-17 23:45:55.621791763 +0000 UTC m=+0.158083413 container create 55a544c384fb599415b45dd3ca96e825cd06af9197289d0b916c961a4cd7f307 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-17T23:45:55.859 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:55 smithi012 podman[31756]: 2024-03-17 23:45:55.858586684 +0000 UTC m=+0.394878338 container init 55a544c384fb599415b45dd3ca96e825cd06af9197289d0b916c961a4cd7f307 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-17T23:45:56.000 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-17T23:45:56.001 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-03-17T23:45:56.020 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-17T23:45:56.020 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-17T23:45:56.021 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-03-17T23:45:56.021 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to enable service . firewalld.service is not available 2024-03-17T23:45:56.045 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-17T23:45:56.046 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-03-17T23:45:56.067 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-17T23:45:56.067 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-17T23:45:56.067 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-03-17T23:45:56.068 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-17T23:45:56.068 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr to start... 2024-03-17T23:45:56.068 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr... 2024-03-17T23:45:56.169 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:55 smithi012 podman[31756]: 2024-03-17 23:45:55.869999065 +0000 UTC m=+0.406290692 container start 55a544c384fb599415b45dd3ca96e825cd06af9197289d0b916c961a4cd7f307 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-17T23:45:56.169 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:55 smithi012 bash[31756]: 55a544c384fb599415b45dd3ca96e825cd06af9197289d0b916c961a4cd7f307 2024-03-17T23:45:56.169 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:55 smithi012 systemd[1]: Started Ceph mgr.x for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:45:56.796 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:56 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:45:56.669+0000 7fa593e75200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-17T23:45:57.111 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:45:56 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3209341608' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-17T23:45:57.111 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:56 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:45:56.800+0000 7fa593e75200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-17T23:45:57.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-17T23:45:57.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-17T23:45:57.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:45:57.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-03-17T23:45:57.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-17T23:45:57.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-17T23:45:57.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-17T23:45:57.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:45:57.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-17T23:45:57.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-17T23:45:57.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-03-17T23:45:57.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-17T23:45:57.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-17T23:45:57.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-03-17T23:45:57.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-17T23:45:57.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-17T23:45:57.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-17T23:45:57.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:45:57.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-17T23:45:57.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-17T23:45:57.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:45:57.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-17T23:45:57.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:45:57.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-17T23:45:57.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-17T23:45:57.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-17T23:45:57.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-17T23:45:57.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-17T23:45:57.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-17T23:45:57.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:45:57.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-17T23:45:57.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-17T23:45:57.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-17T23:45:57.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-17T23:45:57.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-17T23:45:57.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-17T23:45:57.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-17T23:45:57.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-17T23:45:57.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-17T23:45:57.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:45:57.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-17T23:45:57.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:45:57.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-17T23:45:57.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-17T23:45:57.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:45:57.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-17T23:45:57.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-17T23:45:57.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-17T23:45:57.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-17T23:45:57.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-17T23:45:57.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-17T23:45:57.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-03-17T23:45:57.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-17T23:45:57.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-17T23:45:57.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:45:57.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-17T23:45:57.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:45:57.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-03-17T23:45:45.898766+0000", 2024-03-17T23:45:57.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-17T23:45:57.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:45:57.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-17T23:45:57.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-17T23:45:57.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.699+0000 7fe8db400700 1 Processor -- start 2024-03-17T23:45:57.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.699+0000 7fe8db400700 1 -- start start 2024-03-17T23:45:57.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.700+0000 7fe8db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 0x7fe8d407b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:45:57.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.700+0000 7fe8db400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d407b7e0 con 0x7fe8d407cdb0 2024-03-17T23:45:57.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.700+0000 7fe8d8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 0x7fe8d407b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:45:57.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.700+0000 7fe8d8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 0x7fe8d407b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48650/0 (socket says 172.21.15.12:48650) 2024-03-17T23:45:57.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.700+0000 7fe8d8a00700 1 -- 172.21.15.12:0/333921269 learned_addr learned my addr 172.21.15.12:0/333921269 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:45:57.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.701+0000 7fe8d8a00700 1 -- 172.21.15.12:0/333921269 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d407b960 con 0x7fe8d407cdb0 2024-03-17T23:45:57.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.701+0000 7fe8d8a00700 1 --2- 172.21.15.12:0/333921269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 0x7fe8d407b210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80098a0 tx=0x7fe8c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f4ea22f49d4c0c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:45:57.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.701+0000 7fe8d3400700 1 -- 172.21.15.12:0/333921269 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8c8004160 con 0x7fe8d407cdb0 2024-03-17T23:45:57.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.701+0000 7fe8d3400700 1 -- 172.21.15.12:0/333921269 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe8c8004300 con 0x7fe8d407cdb0 2024-03-17T23:45:57.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.701+0000 7fe8d3400700 1 -- 172.21.15.12:0/333921269 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8c8011b50 con 0x7fe8d407cdb0 2024-03-17T23:45:57.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.702+0000 7fe8db400700 1 -- 172.21.15.12:0/333921269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 msgr2=0x7fe8d407b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:45:57.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.702+0000 7fe8db400700 1 --2- 172.21.15.12:0/333921269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 0x7fe8d407b210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80098a0 tx=0x7fe8c8009bb0 comp rx=0 tx=0).stop 2024-03-17T23:45:57.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.702+0000 7fe8db400700 1 -- 172.21.15.12:0/333921269 shutdown_connections 2024-03-17T23:45:57.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.702+0000 7fe8db400700 1 --2- 172.21.15.12:0/333921269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 0x7fe8d407b210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:45:57.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.702+0000 7fe8db400700 1 -- 172.21.15.12:0/333921269 >> 172.21.15.12:0/333921269 conn(0x7fe8d41031f0 msgr2=0x7fe8d4105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:45:57.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.702+0000 7fe8db400700 1 -- 172.21.15.12:0/333921269 shutdown_connections 2024-03-17T23:45:57.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.702+0000 7fe8db400700 1 -- 172.21.15.12:0/333921269 wait complete. 2024-03-17T23:45:57.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.703+0000 7fe8db400700 1 Processor -- start 2024-03-17T23:45:57.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.703+0000 7fe8db400700 1 -- start start 2024-03-17T23:45:57.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.704+0000 7fe8db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 0x7fe8d419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:45:57.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.704+0000 7fe8db400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d410e240 con 0x7fe8d407cdb0 2024-03-17T23:45:57.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.704+0000 7fe8d8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 0x7fe8d419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:45:57.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.704+0000 7fe8d8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 0x7fe8d419de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48654/0 (socket says 172.21.15.12:48654) 2024-03-17T23:45:57.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.704+0000 7fe8d8a00700 1 -- 172.21.15.12:0/3209341608 learned_addr learned my addr 172.21.15.12:0/3209341608 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:45:57.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.704+0000 7fe8d8a00700 1 -- 172.21.15.12:0/3209341608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d419e360 con 0x7fe8d407cdb0 2024-03-17T23:45:57.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.705+0000 7fe8d8a00700 1 --2- 172.21.15.12:0/3209341608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 0x7fe8d419de20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8000c00 tx=0x7fe8c8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:45:57.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.705+0000 7fe8d1600700 1 -- 172.21.15.12:0/3209341608 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8c8003b20 con 0x7fe8d407cdb0 2024-03-17T23:45:57.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.705+0000 7fe8d1600700 1 -- 172.21.15.12:0/3209341608 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe8c8003cc0 con 0x7fe8d407cdb0 2024-03-17T23:45:57.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.705+0000 7fe8d1600700 1 -- 172.21.15.12:0/3209341608 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8c8024910 con 0x7fe8d407cdb0 2024-03-17T23:45:57.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.705+0000 7fe8db400700 1 -- 172.21.15.12:0/3209341608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8d419e5f0 con 0x7fe8d407cdb0 2024-03-17T23:45:57.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.705+0000 7fe8db400700 1 -- 172.21.15.12:0/3209341608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8d41a12e0 con 0x7fe8d407cdb0 2024-03-17T23:45:57.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.706+0000 7fe8d1600700 1 -- 172.21.15.12:0/3209341608 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe8c8022020 con 0x7fe8d407cdb0 2024-03-17T23:45:57.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.706+0000 7fe8d1600700 1 -- 172.21.15.12:0/3209341608 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe8c8011e00 con 0x7fe8d407cdb0 2024-03-17T23:45:57.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.706+0000 7fe8db400700 1 -- 172.21.15.12:0/3209341608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8d4063900 con 0x7fe8d407cdb0 2024-03-17T23:45:57.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.709+0000 7fe8d1600700 1 -- 172.21.15.12:0/3209341608 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe8c802d3f0 con 0x7fe8d407cdb0 2024-03-17T23:45:57.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.789+0000 7fe8db400700 1 -- 172.21.15.12:0/3209341608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fe8d41a14b0 con 0x7fe8d407cdb0 2024-03-17T23:45:57.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.790+0000 7fe8d1600700 1 -- 172.21.15.12:0/3209341608 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fe8c801acd0 con 0x7fe8d407cdb0 2024-03-17T23:45:57.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.791+0000 7fe8db400700 1 -- 172.21.15.12:0/3209341608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 msgr2=0x7fe8d419de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:45:57.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.791+0000 7fe8db400700 1 --2- 172.21.15.12:0/3209341608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 0x7fe8d419de20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8000c00 tx=0x7fe8c8011c10 comp rx=0 tx=0).stop 2024-03-17T23:45:57.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.791+0000 7fe8db400700 1 -- 172.21.15.12:0/3209341608 shutdown_connections 2024-03-17T23:45:57.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.791+0000 7fe8db400700 1 --2- 172.21.15.12:0/3209341608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8d407cdb0 0x7fe8d419de20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:45:57.465 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.791+0000 7fe8db400700 1 -- 172.21.15.12:0/3209341608 >> 172.21.15.12:0/3209341608 conn(0x7fe8d41031f0 msgr2=0x7fe8d4103d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:45:57.465 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.791+0000 7fe8db400700 1 -- 172.21.15.12:0/3209341608 shutdown_connections 2024-03-17T23:45:57.465 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:45:56.791+0000 7fe8db400700 1 -- 172.21.15.12:0/3209341608 wait complete. 2024-03-17T23:45:57.465 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (1/15)... 2024-03-17T23:45:57.784 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:57 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:45:57.657+0000 7fa593e75200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-17T23:45:58.096 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:57 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:45:57.783+0000 7fa593e75200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-17T23:45:58.096 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:57 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:45:57.966+0000 7fa593e75200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-17T23:45:58.096 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:58 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:45:58.094+0000 7fa593e75200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-17T23:45:58.421 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:58 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:45:58.207+0000 7fa593e75200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-17T23:45:58.703 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:58 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:45:58.553+0000 7fa593e75200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-17T23:45:58.703 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:58 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:45:58.701+0000 7fa593e75200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-17T23:45:58.955 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:58 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:45:58.823+0000 7fa593e75200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-17T23:45:58.956 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:58 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:45:58.954+0000 7fa593e75200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-17T23:45:59.418 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:45:59 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:45:59.181+0000 7fa593e75200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-17T23:46:00.554 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:00 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2686780044' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-17T23:46:00.853 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:00 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:00.610+0000 7fa593e75200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-17T23:46:00.853 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:00 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:00.790+0000 7fa593e75200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-17T23:46:00.862 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-17T23:46:00.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-17T23:46:00.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:46:00.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-03-17T23:46:00.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-17T23:46:00.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-17T23:46:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-17T23:46:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-17T23:46:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-17T23:46:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-03-17T23:46:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-17T23:46:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-17T23:46:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-03-17T23:46:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-17T23:46:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-17T23:46:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-17T23:46:00.874 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:46:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-17T23:46:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-17T23:46:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-17T23:46:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:46:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-17T23:46:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-17T23:46:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-17T23:46:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-17T23:46:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-17T23:46:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-17T23:46:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:00.875 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-17T23:46:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-17T23:46:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-17T23:46:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-17T23:46:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-17T23:46:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-17T23:46:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-17T23:46:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-17T23:46:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-17T23:46:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-17T23:46:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:46:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-17T23:46:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-17T23:46:00.877 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:00.877 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-17T23:46:00.877 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-17T23:46:00.877 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-17T23:46:00.877 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-17T23:46:00.877 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-17T23:46:00.877 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-17T23:46:00.877 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-03-17T23:46:00.877 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-17T23:46:00.877 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-17T23:46:00.877 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:00.878 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-17T23:46:00.878 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:46:00.878 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-03-17T23:45:45.898766+0000", 2024-03-17T23:46:00.878 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-17T23:46:00.878 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:00.878 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-17T23:46:00.878 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-17T23:46:00.878 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.090+0000 7fbbbc600700 1 Processor -- start 2024-03-17T23:46:00.881 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.090+0000 7fbbbc600700 1 -- start start 2024-03-17T23:46:00.881 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.091+0000 7fbbbc600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 0x7fbbb410a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:00.881 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.091+0000 7fbbbc600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb410a890 con 0x7fbbb4109f10 2024-03-17T23:46:00.881 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.091+0000 7fbbb9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 0x7fbbb410a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:00.882 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.091+0000 7fbbb9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 0x7fbbb410a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48656/0 (socket says 172.21.15.12:48656) 2024-03-17T23:46:00.882 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.091+0000 7fbbb9c00700 1 -- 172.21.15.12:0/2832628167 learned_addr learned my addr 172.21.15.12:0/2832628167 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:00.882 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.091+0000 7fbbb9c00700 1 -- 172.21.15.12:0/2832628167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb410b0f0 con 0x7fbbb4109f10 2024-03-17T23:46:00.882 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.092+0000 7fbbb9c00700 1 --2- 172.21.15.12:0/2832628167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 0x7fbbb410a350 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbba80098a0 tx=0x7fbba8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ada57137dc6283 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:00.882 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.092+0000 7fbbb3e00700 1 -- 172.21.15.12:0/2832628167 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbba8004160 con 0x7fbbb4109f10 2024-03-17T23:46:00.882 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.092+0000 7fbbb3e00700 1 -- 172.21.15.12:0/2832628167 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbba8004300 con 0x7fbbb4109f10 2024-03-17T23:46:00.882 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.092+0000 7fbbb3e00700 1 -- 172.21.15.12:0/2832628167 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbba8011b50 con 0x7fbbb4109f10 2024-03-17T23:46:00.882 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.093+0000 7fbbbc600700 1 -- 172.21.15.12:0/2832628167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 msgr2=0x7fbbb410a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:00.882 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.093+0000 7fbbbc600700 1 --2- 172.21.15.12:0/2832628167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 0x7fbbb410a350 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbba80098a0 tx=0x7fbba8009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:00.882 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.093+0000 7fbbbc600700 1 -- 172.21.15.12:0/2832628167 shutdown_connections 2024-03-17T23:46:00.882 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.093+0000 7fbbbc600700 1 --2- 172.21.15.12:0/2832628167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 0x7fbbb410a350 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:00.882 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.093+0000 7fbbbc600700 1 -- 172.21.15.12:0/2832628167 >> 172.21.15.12:0/2832628167 conn(0x7fbbb407d370 msgr2=0x7fbbb407d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:00.882 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.093+0000 7fbbbc600700 1 -- 172.21.15.12:0/2832628167 shutdown_connections 2024-03-17T23:46:00.883 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.093+0000 7fbbbc600700 1 -- 172.21.15.12:0/2832628167 wait complete. 2024-03-17T23:46:00.883 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.094+0000 7fbbbc600700 1 Processor -- start 2024-03-17T23:46:00.883 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.094+0000 7fbbbc600700 1 -- start start 2024-03-17T23:46:00.883 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.094+0000 7fbbbc600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 0x7fbbb419e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:00.883 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.094+0000 7fbbbc600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb410e3b0 con 0x7fbbb4109f10 2024-03-17T23:46:00.883 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.095+0000 7fbbb9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 0x7fbbb419e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:00.883 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.095+0000 7fbbb9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 0x7fbbb419e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48660/0 (socket says 172.21.15.12:48660) 2024-03-17T23:46:00.883 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.095+0000 7fbbb9c00700 1 -- 172.21.15.12:0/2686780044 learned_addr learned my addr 172.21.15.12:0/2686780044 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:00.883 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.095+0000 7fbbb9c00700 1 -- 172.21.15.12:0/2686780044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb419e780 con 0x7fbbb4109f10 2024-03-17T23:46:00.883 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.095+0000 7fbbb9c00700 1 --2- 172.21.15.12:0/2686780044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 0x7fbbb419e240 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbba8000c00 tx=0x7fbba8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:00.883 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.096+0000 7fbbb2000700 1 -- 172.21.15.12:0/2686780044 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbba8003b20 con 0x7fbbb4109f10 2024-03-17T23:46:00.884 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.096+0000 7fbbb2000700 1 -- 172.21.15.12:0/2686780044 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbba8003cc0 con 0x7fbbb4109f10 2024-03-17T23:46:00.884 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.096+0000 7fbbbc600700 1 -- 172.21.15.12:0/2686780044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb419ea10 con 0x7fbbb4109f10 2024-03-17T23:46:00.884 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.096+0000 7fbbb2000700 1 -- 172.21.15.12:0/2686780044 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbba8024910 con 0x7fbbb4109f10 2024-03-17T23:46:00.884 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.096+0000 7fbbbc600700 1 -- 172.21.15.12:0/2686780044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb41a1700 con 0x7fbbb4109f10 2024-03-17T23:46:00.884 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.096+0000 7fbbb2000700 1 -- 172.21.15.12:0/2686780044 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbba8022020 con 0x7fbbb4109f10 2024-03-17T23:46:00.884 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.096+0000 7fbbb2000700 1 -- 172.21.15.12:0/2686780044 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbba8011e00 con 0x7fbbb4109f10 2024-03-17T23:46:00.884 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.097+0000 7fbbbc600700 1 -- 172.21.15.12:0/2686780044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbb410e530 con 0x7fbbb4109f10 2024-03-17T23:46:00.885 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.101+0000 7fbbb2000700 1 -- 172.21.15.12:0/2686780044 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fbba8042dd0 con 0x7fbbb4109f10 2024-03-17T23:46:00.885 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.179+0000 7fbbbc600700 1 -- 172.21.15.12:0/2686780044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fbbb402db70 con 0x7fbbb4109f10 2024-03-17T23:46:00.885 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.180+0000 7fbbb2000700 1 -- 172.21.15.12:0/2686780044 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fbba8031020 con 0x7fbbb4109f10 2024-03-17T23:46:00.885 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.181+0000 7fbbbc600700 1 -- 172.21.15.12:0/2686780044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 msgr2=0x7fbbb419e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:00.885 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.181+0000 7fbbbc600700 1 --2- 172.21.15.12:0/2686780044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 0x7fbbb419e240 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbba8000c00 tx=0x7fbba8011c10 comp rx=0 tx=0).stop 2024-03-17T23:46:00.885 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.181+0000 7fbbbc600700 1 -- 172.21.15.12:0/2686780044 shutdown_connections 2024-03-17T23:46:00.885 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.181+0000 7fbbbc600700 1 --2- 172.21.15.12:0/2686780044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb4109f10 0x7fbbb419e240 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:00.885 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.181+0000 7fbbbc600700 1 -- 172.21.15.12:0/2686780044 >> 172.21.15.12:0/2686780044 conn(0x7fbbb407d370 msgr2=0x7fbbb41076d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:00.885 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.181+0000 7fbbbc600700 1 -- 172.21.15.12:0/2686780044 shutdown_connections 2024-03-17T23:46:00.885 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:00.181+0000 7fbbbc600700 1 -- 172.21.15.12:0/2686780044 wait complete. 2024-03-17T23:46:00.885 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (2/15)... 2024-03-17T23:46:01.608 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:01 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:01.219+0000 7fa593e75200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-17T23:46:01.609 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:01 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:01.342+0000 7fa593e75200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-17T23:46:01.860 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:01 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:01.607+0000 7fa593e75200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:46:02.168 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:01 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:01.859+0000 7fa593e75200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-17T23:46:02.569 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:02 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:02.244+0000 7fa593e75200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:46:02.569 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:02 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:02.567+0000 7fa593e75200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-17T23:46:03.160 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:02 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:02.882+0000 7fa593e75200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-17T23:46:03.909 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:03 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4069021549' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-17T23:46:04.217 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:03 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:03.976+0000 7fa593e75200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-17T23:46:04.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-17T23:46:04.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-17T23:46:04.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:46:04.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-03-17T23:46:04.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-17T23:46:04.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-17T23:46:04.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-17T23:46:04.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:04.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-17T23:46:04.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-17T23:46:04.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-03-17T23:46:04.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-17T23:46:04.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-17T23:46:04.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-03-17T23:46:04.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-17T23:46:04.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-17T23:46:04.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-17T23:46:04.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:46:04.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-17T23:46:04.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-17T23:46:04.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:04.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-17T23:46:04.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-17T23:46:04.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-17T23:46:04.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-17T23:46:04.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-03-17T23:46:04.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-17T23:46:04.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-17T23:46:04.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:04.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-17T23:46:04.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:46:04.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-03-17T23:45:45.898766+0000", 2024-03-17T23:46:04.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-17T23:46:04.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:04.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-17T23:46:04.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-17T23:46:04.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.451+0000 7fb194800700 1 Processor -- start 2024-03-17T23:46:04.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.452+0000 7fb194800700 1 -- start start 2024-03-17T23:46:04.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.452+0000 7fb194800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 0x7fb18c10ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:04.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.452+0000 7fb194800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c0759e0 con 0x7fb18c108720 2024-03-17T23:46:04.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.453+0000 7fb191e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 0x7fb18c10ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:04.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.453+0000 7fb191e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 0x7fb18c10ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48666/0 (socket says 172.21.15.12:48666) 2024-03-17T23:46:04.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.453+0000 7fb191e00700 1 -- 172.21.15.12:0/1062779677 learned_addr learned my addr 172.21.15.12:0/1062779677 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:04.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.453+0000 7fb191e00700 1 -- 172.21.15.12:0/1062779677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18c10b050 con 0x7fb18c108720 2024-03-17T23:46:04.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.453+0000 7fb191e00700 1 --2- 172.21.15.12:0/1062779677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 0x7fb18c10ab10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb1800098a0 tx=0x7fb180009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9324f6a2731f3fa server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:04.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.454+0000 7fb190a00700 1 -- 172.21.15.12:0/1062779677 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb180004160 con 0x7fb18c108720 2024-03-17T23:46:04.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.454+0000 7fb190a00700 1 -- 172.21.15.12:0/1062779677 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb180004300 con 0x7fb18c108720 2024-03-17T23:46:04.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.454+0000 7fb190a00700 1 -- 172.21.15.12:0/1062779677 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb180011b50 con 0x7fb18c108720 2024-03-17T23:46:04.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.454+0000 7fb194800700 1 -- 172.21.15.12:0/1062779677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 msgr2=0x7fb18c10ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:04.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.454+0000 7fb194800700 1 --2- 172.21.15.12:0/1062779677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 0x7fb18c10ab10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb1800098a0 tx=0x7fb180009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:04.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.455+0000 7fb194800700 1 -- 172.21.15.12:0/1062779677 shutdown_connections 2024-03-17T23:46:04.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.455+0000 7fb194800700 1 --2- 172.21.15.12:0/1062779677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 0x7fb18c10ab10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:04.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.455+0000 7fb194800700 1 -- 172.21.15.12:0/1062779677 >> 172.21.15.12:0/1062779677 conn(0x7fb18c102130 msgr2=0x7fb18c104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:04.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.455+0000 7fb194800700 1 -- 172.21.15.12:0/1062779677 shutdown_connections 2024-03-17T23:46:04.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.455+0000 7fb194800700 1 -- 172.21.15.12:0/1062779677 wait complete. 2024-03-17T23:46:04.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.456+0000 7fb194800700 1 Processor -- start 2024-03-17T23:46:04.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.456+0000 7fb194800700 1 -- start start 2024-03-17T23:46:04.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.457+0000 7fb194800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 0x7fb18c19bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:04.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.457+0000 7fb194800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c10c020 con 0x7fb18c108720 2024-03-17T23:46:04.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.457+0000 7fb191e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 0x7fb18c19bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:04.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.457+0000 7fb191e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 0x7fb18c19bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48668/0 (socket says 172.21.15.12:48668) 2024-03-17T23:46:04.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.457+0000 7fb191e00700 1 -- 172.21.15.12:0/4069021549 learned_addr learned my addr 172.21.15.12:0/4069021549 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.457+0000 7fb191e00700 1 -- 172.21.15.12:0/4069021549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18c19c140 con 0x7fb18c108720 2024-03-17T23:46:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.458+0000 7fb191e00700 1 --2- 172.21.15.12:0/4069021549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 0x7fb18c19bc00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb180011b60 tx=0x7fb180011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.458+0000 7fb186a00700 1 -- 172.21.15.12:0/4069021549 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb180003b20 con 0x7fb18c108720 2024-03-17T23:46:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.458+0000 7fb186a00700 1 -- 172.21.15.12:0/4069021549 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb180003cc0 con 0x7fb18c108720 2024-03-17T23:46:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.458+0000 7fb186a00700 1 -- 172.21.15.12:0/4069021549 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb180024840 con 0x7fb18c108720 2024-03-17T23:46:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.458+0000 7fb194800700 1 -- 172.21.15.12:0/4069021549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18c19c3d0 con 0x7fb18c108720 2024-03-17T23:46:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.458+0000 7fb194800700 1 -- 172.21.15.12:0/4069021549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb18c19f070 con 0x7fb18c108720 2024-03-17T23:46:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.459+0000 7fb186a00700 1 -- 172.21.15.12:0/4069021549 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb180022020 con 0x7fb18c108720 2024-03-17T23:46:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.459+0000 7fb186a00700 1 -- 172.21.15.12:0/4069021549 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb180011e00 con 0x7fb18c108720 2024-03-17T23:46:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.459+0000 7fb194800700 1 -- 172.21.15.12:0/4069021549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb18c063900 con 0x7fb18c108720 2024-03-17T23:46:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.462+0000 7fb186a00700 1 -- 172.21.15.12:0/4069021549 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb180043dd0 con 0x7fb18c108720 2024-03-17T23:46:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.541+0000 7fb194800700 1 -- 172.21.15.12:0/4069021549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fb18c19f690 con 0x7fb18c108720 2024-03-17T23:46:04.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.542+0000 7fb186a00700 1 -- 172.21.15.12:0/4069021549 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7fb180032030 con 0x7fb18c108720 2024-03-17T23:46:04.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.543+0000 7fb194800700 1 -- 172.21.15.12:0/4069021549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 msgr2=0x7fb18c19bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:04.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.543+0000 7fb194800700 1 --2- 172.21.15.12:0/4069021549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 0x7fb18c19bc00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb180011b60 tx=0x7fb180011c40 comp rx=0 tx=0).stop 2024-03-17T23:46:04.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.543+0000 7fb194800700 1 -- 172.21.15.12:0/4069021549 shutdown_connections 2024-03-17T23:46:04.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.543+0000 7fb194800700 1 --2- 172.21.15.12:0/4069021549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb18c108720 0x7fb18c19bc00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:04.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.543+0000 7fb194800700 1 -- 172.21.15.12:0/4069021549 >> 172.21.15.12:0/4069021549 conn(0x7fb18c102130 msgr2=0x7fb18c102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:04.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.543+0000 7fb194800700 1 -- 172.21.15.12:0/4069021549 shutdown_connections 2024-03-17T23:46:04.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:03.543+0000 7fb194800700 1 -- 172.21.15.12:0/4069021549 wait complete. 2024-03-17T23:46:04.258 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (3/15)... 2024-03-17T23:46:05.123 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:04 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:04.698+0000 7fa593e75200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-17T23:46:05.124 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:04 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:04.831+0000 7fa593e75200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-17T23:46:05.419 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:05 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:05.122+0000 7fa593e75200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-17T23:46:05.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:05 smithi012 ceph-mon[31541]: Activating manager daemon x 2024-03-17T23:46:05.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:05 smithi012 ceph-mon[31541]: mgrmap e2: x(active, starting, since 0.00639833s) 2024-03-17T23:46:05.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:05 smithi012 ceph-mon[31541]: from='mgr.14100 172.21.15.12:0/1328434495' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-17T23:46:05.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:05 smithi012 ceph-mon[31541]: from='mgr.14100 172.21.15.12:0/1328434495' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-17T23:46:05.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:05 smithi012 ceph-mon[31541]: from='mgr.14100 172.21.15.12:0/1328434495' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-17T23:46:05.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:05 smithi012 ceph-mon[31541]: from='mgr.14100 172.21.15.12:0/1328434495' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:46:05.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:05 smithi012 ceph-mon[31541]: from='mgr.14100 172.21.15.12:0/1328434495' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-17T23:46:05.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:05 smithi012 ceph-mon[31541]: Manager daemon x is now available 2024-03-17T23:46:05.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:05 smithi012 ceph-mon[31541]: from='mgr.14100 172.21.15.12:0/1328434495' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-17T23:46:05.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:05 smithi012 ceph-mon[31541]: from='mgr.14100 172.21.15.12:0/1328434495' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-17T23:46:05.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:05 smithi012 ceph-mon[31541]: from='mgr.14100 172.21.15.12:0/1328434495' entity='mgr.x' 2024-03-17T23:46:05.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:05 smithi012 ceph-mon[31541]: from='mgr.14100 172.21.15.12:0/1328434495' entity='mgr.x' 2024-03-17T23:46:06.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:06 smithi012 ceph-mon[31541]: from='mgr.14100 172.21.15.12:0/1328434495' entity='mgr.x' 2024-03-17T23:46:06.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:06 smithi012 ceph-mon[31541]: mgrmap e3: x(active, since 1.0238s) 2024-03-17T23:46:07.890 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-17T23:46:07.891 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-17T23:46:07.892 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:46:07.892 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-03-17T23:46:07.892 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-17T23:46:07.892 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-17T23:46:07.892 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-17T23:46:07.892 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:07.892 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-17T23:46:07.893 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-17T23:46:07.894 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-03-17T23:46:07.894 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-17T23:46:07.894 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-17T23:46:07.894 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-03-17T23:46:07.894 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-17T23:46:07.894 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-17T23:46:07.894 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-17T23:46:07.894 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:46:07.894 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-17T23:46:07.894 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-17T23:46:07.894 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:07.895 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-17T23:46:07.896 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:46:07.896 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-17T23:46:07.896 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-17T23:46:07.896 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-17T23:46:07.896 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-17T23:46:07.896 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-17T23:46:07.896 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-17T23:46:07.896 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:07.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-17T23:46:07.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-17T23:46:07.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-17T23:46:07.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-17T23:46:07.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-17T23:46:07.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-17T23:46:07.897 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-17T23:46:07.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-17T23:46:07.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-17T23:46:07.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:07.898 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-17T23:46:07.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:46:07.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-17T23:46:07.899 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-17T23:46:07.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:07.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-17T23:46:07.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-17T23:46:07.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-17T23:46:07.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-03-17T23:45:45.898766+0000", 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-17T23:46:07.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.862+0000 7febdc000700 1 Processor -- start 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.863+0000 7febdc000700 1 -- start start 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.863+0000 7febdc000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 0x7febd4108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.863+0000 7febdc000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7febd4108640 con 0x7febd4107c50 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.863+0000 7febd9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 0x7febd4108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.863+0000 7febd9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 0x7febd4108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54966/0 (socket says 172.21.15.12:54966) 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.864+0000 7febd9600700 1 -- 172.21.15.12:0/509426934 learned_addr learned my addr 172.21.15.12:0/509426934 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.864+0000 7febd9600700 1 -- 172.21.15.12:0/509426934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd4108e50 con 0x7febd4107c50 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.864+0000 7febd9600700 1 --2- 172.21.15.12:0/509426934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 0x7febd4108070 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7febc8009e80 tx=0x7febc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd1cf6911ca04bf server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.865+0000 7febd3e00700 1 -- 172.21.15.12:0/509426934 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febc8017070 con 0x7febd4107c50 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.865+0000 7febd3e00700 1 -- 172.21.15.12:0/509426934 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7febc800b8c0 con 0x7febd4107c50 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.865+0000 7febd3e00700 1 -- 172.21.15.12:0/509426934 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febc80045c0 con 0x7febd4107c50 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.865+0000 7febdc000700 1 -- 172.21.15.12:0/509426934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 msgr2=0x7febd4108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.865+0000 7febdc000700 1 --2- 172.21.15.12:0/509426934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 0x7febd4108070 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7febc8009e80 tx=0x7febc800b330 comp rx=0 tx=0).stop 2024-03-17T23:46:07.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.866+0000 7febdc000700 1 -- 172.21.15.12:0/509426934 shutdown_connections 2024-03-17T23:46:07.906 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.866+0000 7febdc000700 1 --2- 172.21.15.12:0/509426934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 0x7febd4108070 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:07.906 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.866+0000 7febdc000700 1 -- 172.21.15.12:0/509426934 >> 172.21.15.12:0/509426934 conn(0x7febd41031f0 msgr2=0x7febd4105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:07.906 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.866+0000 7febdc000700 1 -- 172.21.15.12:0/509426934 shutdown_connections 2024-03-17T23:46:07.906 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.866+0000 7febdc000700 1 -- 172.21.15.12:0/509426934 wait complete. 2024-03-17T23:46:07.906 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.867+0000 7febdc000700 1 Processor -- start 2024-03-17T23:46:07.909 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.867+0000 7febdc000700 1 -- start start 2024-03-17T23:46:07.909 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.867+0000 7febdc000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 0x7febd419bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:07.909 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.867+0000 7febdc000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7febd410c200 con 0x7febd4107c50 2024-03-17T23:46:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.868+0000 7febd9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 0x7febd419bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.868+0000 7febd9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 0x7febd419bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54976/0 (socket says 172.21.15.12:54976) 2024-03-17T23:46:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.868+0000 7febd9600700 1 -- 172.21.15.12:0/3912438866 learned_addr learned my addr 172.21.15.12:0/3912438866 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.868+0000 7febd9600700 1 -- 172.21.15.12:0/3912438866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd419c160 con 0x7febd4107c50 2024-03-17T23:46:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.868+0000 7febd9600700 1 --2- 172.21.15.12:0/3912438866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 0x7febd419bc20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7febc8009530 tx=0x7febc800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.868+0000 7febd2000700 1 -- 172.21.15.12:0/3912438866 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febc8020020 con 0x7febd4107c50 2024-03-17T23:46:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.868+0000 7febd2000700 1 -- 172.21.15.12:0/3912438866 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7febc8003b50 con 0x7febd4107c50 2024-03-17T23:46:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.869+0000 7febd2000700 1 -- 172.21.15.12:0/3912438866 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febc8017040 con 0x7febd4107c50 2024-03-17T23:46:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.869+0000 7febdc000700 1 -- 172.21.15.12:0/3912438866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febd419c3f0 con 0x7febd4107c50 2024-03-17T23:46:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.869+0000 7febdc000700 1 -- 172.21.15.12:0/3912438866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febd419f0e0 con 0x7febd4107c50 2024-03-17T23:46:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.870+0000 7febdc000700 1 -- 172.21.15.12:0/3912438866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febd4063900 con 0x7febd4107c50 2024-03-17T23:46:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.870+0000 7febd2000700 1 -- 172.21.15.12:0/3912438866 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 3) ==== 45989+0+0 (secure 0 0 0) 0x7febc8023020 con 0x7febd4107c50 2024-03-17T23:46:07.910 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.870+0000 7febd2000700 1 --2- 172.21.15.12:0/3912438866 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7febbc038e00 0x7febbc03b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.870+0000 7febd2000700 1 -- 172.21.15.12:0/3912438866 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7febc800f050 con 0x7febd4107c50 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.871+0000 7febd8c00700 1 --2- 172.21.15.12:0/3912438866 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7febbc038e00 0x7febbc03b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.871+0000 7febd8c00700 1 --2- 172.21.15.12:0/3912438866 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7febbc038e00 0x7febbc03b2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7febc0006fd0 tx=0x7febc0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:06.877+0000 7febd2000700 1 -- 172.21.15.12:0/3912438866 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7febc802a020 con 0x7febd4107c50 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:07.189+0000 7febd2000700 1 -- 172.21.15.12:0/3912438866 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7febc8012430 con 0x7febd4107c50 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:07.190+0000 7febdc000700 1 -- 172.21.15.12:0/3912438866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7febd419f730 con 0x7febd4107c50 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:07.192+0000 7febd2000700 1 -- 172.21.15.12:0/3912438866 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7febc802d350 con 0x7febd4107c50 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:07.195+0000 7febdc000700 1 -- 172.21.15.12:0/3912438866 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7febbc038e00 msgr2=0x7febbc03b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:07.195+0000 7febdc000700 1 --2- 172.21.15.12:0/3912438866 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7febbc038e00 0x7febbc03b2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7febc0006fd0 tx=0x7febc0006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:07.195+0000 7febdc000700 1 -- 172.21.15.12:0/3912438866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 msgr2=0x7febd419bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:07.195+0000 7febdc000700 1 --2- 172.21.15.12:0/3912438866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 0x7febd419bc20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7febc8009530 tx=0x7febc800bd50 comp rx=0 tx=0).stop 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:07.195+0000 7febdc000700 1 -- 172.21.15.12:0/3912438866 shutdown_connections 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:07.195+0000 7febdc000700 1 --2- 172.21.15.12:0/3912438866 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7febbc038e00 0x7febbc03b2c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:07.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:07.195+0000 7febdc000700 1 --2- 172.21.15.12:0/3912438866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febd4107c50 0x7febd419bc20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:07.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:07.195+0000 7febdc000700 1 -- 172.21.15.12:0/3912438866 >> 172.21.15.12:0/3912438866 conn(0x7febd41031f0 msgr2=0x7febd4103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:07.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:07.196+0000 7febdc000700 1 -- 172.21.15.12:0/3912438866 shutdown_connections 2024-03-17T23:46:07.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:07.196+0000 7febdc000700 1 -- 172.21.15.12:0/3912438866 wait complete. 2024-03-17T23:46:07.912 INFO:teuthology.orchestra.run.smithi012.stdout:mgr is available 2024-03-17T23:46:08.492 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:08 smithi012 ceph-mon[31541]: mgrmap e4: x(active, since 2s) 2024-03-17T23:46:08.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:08 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3912438866' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-17T23:46:09.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:09 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2869776393' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-17T23:46:09.589 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-17T23:46:09.589 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [global] 2024-03-17T23:46:09.589 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout fsid = 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:46:09.589 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-17T23:46:09.589 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-17T23:46:09.602 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-17T23:46:09.602 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-17T23:46:09.602 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-17T23:46:09.602 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-17T23:46:09.602 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-17T23:46:09.602 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [osd] 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.531+0000 7fe0b4400700 1 Processor -- start 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.531+0000 7fe0b4400700 1 -- start start 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.532+0000 7fe0b4400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 0x7fe0ac07c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.532+0000 7fe0b4400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ac07c910 con 0x7fe0ac07dee0 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.532+0000 7fe0b1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 0x7fe0ac07c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.532+0000 7fe0b1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 0x7fe0ac07c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54984/0 (socket says 172.21.15.12:54984) 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.532+0000 7fe0b1a00700 1 -- 172.21.15.12:0/2761908256 learned_addr learned my addr 172.21.15.12:0/2761908256 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:09.603 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.533+0000 7fe0b1a00700 1 -- 172.21.15.12:0/2761908256 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0ac07ca90 con 0x7fe0ac07dee0 2024-03-17T23:46:09.604 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.533+0000 7fe0b1a00700 1 --2- 172.21.15.12:0/2761908256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 0x7fe0ac07c340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe0a00098a0 tx=0x7fe0a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c700fd430b6a79 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:09.604 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.533+0000 7fe0abe00700 1 -- 172.21.15.12:0/2761908256 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0a0004160 con 0x7fe0ac07dee0 2024-03-17T23:46:09.604 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.533+0000 7fe0abe00700 1 -- 172.21.15.12:0/2761908256 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe0a0004300 con 0x7fe0ac07dee0 2024-03-17T23:46:09.604 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.533+0000 7fe0abe00700 1 -- 172.21.15.12:0/2761908256 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0a0011b50 con 0x7fe0ac07dee0 2024-03-17T23:46:09.604 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.534+0000 7fe0b4400700 1 -- 172.21.15.12:0/2761908256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 msgr2=0x7fe0ac07c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:09.604 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.534+0000 7fe0b4400700 1 --2- 172.21.15.12:0/2761908256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 0x7fe0ac07c340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe0a00098a0 tx=0x7fe0a0009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:09.604 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.534+0000 7fe0b4400700 1 -- 172.21.15.12:0/2761908256 shutdown_connections 2024-03-17T23:46:09.604 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.534+0000 7fe0b4400700 1 --2- 172.21.15.12:0/2761908256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 0x7fe0ac07c340 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:09.604 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.534+0000 7fe0b4400700 1 -- 172.21.15.12:0/2761908256 >> 172.21.15.12:0/2761908256 conn(0x7fe0ac103260 msgr2=0x7fe0ac105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:09.604 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.534+0000 7fe0b4400700 1 -- 172.21.15.12:0/2761908256 shutdown_connections 2024-03-17T23:46:09.604 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.534+0000 7fe0b4400700 1 -- 172.21.15.12:0/2761908256 wait complete. 2024-03-17T23:46:09.604 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.536+0000 7fe0b4400700 1 Processor -- start 2024-03-17T23:46:09.604 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.536+0000 7fe0b4400700 1 -- start start 2024-03-17T23:46:09.605 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.536+0000 7fe0b4400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 0x7fe0ac19e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:09.605 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.536+0000 7fe0b4400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ac10e2c0 con 0x7fe0ac07dee0 2024-03-17T23:46:09.605 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.536+0000 7fe0b1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 0x7fe0ac19e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:09.605 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.536+0000 7fe0b1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 0x7fe0ac19e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54996/0 (socket says 172.21.15.12:54996) 2024-03-17T23:46:09.605 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.536+0000 7fe0b1a00700 1 -- 172.21.15.12:0/2869776393 learned_addr learned my addr 172.21.15.12:0/2869776393 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:09.605 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.537+0000 7fe0b1a00700 1 -- 172.21.15.12:0/2869776393 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0ac19e6c0 con 0x7fe0ac07dee0 2024-03-17T23:46:09.605 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.537+0000 7fe0b1a00700 1 --2- 172.21.15.12:0/2869776393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 0x7fe0ac19e180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0011670 tx=0x7fe0a0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:09.605 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.537+0000 7fe0aa000700 1 -- 172.21.15.12:0/2869776393 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0a00039c0 con 0x7fe0ac07dee0 2024-03-17T23:46:09.605 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.537+0000 7fe0aa000700 1 -- 172.21.15.12:0/2869776393 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe0a0011e50 con 0x7fe0ac07dee0 2024-03-17T23:46:09.605 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.537+0000 7fe0aa000700 1 -- 172.21.15.12:0/2869776393 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0a001a5c0 con 0x7fe0ac07dee0 2024-03-17T23:46:09.605 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.537+0000 7fe0b4400700 1 -- 172.21.15.12:0/2869776393 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0ac19e950 con 0x7fe0ac07dee0 2024-03-17T23:46:09.606 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.538+0000 7fe0b4400700 1 -- 172.21.15.12:0/2869776393 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0ac1a1640 con 0x7fe0ac07dee0 2024-03-17T23:46:09.606 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.539+0000 7fe0aa000700 1 -- 172.21.15.12:0/2869776393 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fe0a0024430 con 0x7fe0ac07dee0 2024-03-17T23:46:09.606 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.539+0000 7fe0b4400700 1 -- 172.21.15.12:0/2869776393 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0ac063900 con 0x7fe0ac07dee0 2024-03-17T23:46:09.606 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.539+0000 7fe0aa000700 1 --2- 172.21.15.12:0/2869776393 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe094038ed0 0x7fe09403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:09.606 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.539+0000 7fe0aa000700 1 -- 172.21.15.12:0/2869776393 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe0a004d0c0 con 0x7fe0ac07dee0 2024-03-17T23:46:09.606 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.540+0000 7fe0b1000700 1 --2- 172.21.15.12:0/2869776393 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe094038ed0 0x7fe09403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:09.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.540+0000 7fe0b1000700 1 --2- 172.21.15.12:0/2869776393 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe094038ed0 0x7fe09403b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe098006fd0 tx=0x7fe098006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:09.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.545+0000 7fe0aa000700 1 -- 172.21.15.12:0/2869776393 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe0a002d3f0 con 0x7fe0ac07dee0 2024-03-17T23:46:09.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.771+0000 7fe0b4400700 1 -- 172.21.15.12:0/2869776393 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fe0ac1a2990 con 0x7fe0ac07dee0 2024-03-17T23:46:09.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.772+0000 7fe0aa000700 1 -- 172.21.15.12:0/2869776393 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fe0a004f1e0 con 0x7fe0ac07dee0 2024-03-17T23:46:09.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.775+0000 7fe0b4400700 1 -- 172.21.15.12:0/2869776393 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe094038ed0 msgr2=0x7fe09403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:09.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.775+0000 7fe0b4400700 1 --2- 172.21.15.12:0/2869776393 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe094038ed0 0x7fe09403b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe098006fd0 tx=0x7fe098006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:09.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.775+0000 7fe0b4400700 1 -- 172.21.15.12:0/2869776393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 msgr2=0x7fe0ac19e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:09.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.775+0000 7fe0b4400700 1 --2- 172.21.15.12:0/2869776393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 0x7fe0ac19e180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0011670 tx=0x7fe0a0011b60 comp rx=0 tx=0).stop 2024-03-17T23:46:09.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.775+0000 7fe0b4400700 1 -- 172.21.15.12:0/2869776393 shutdown_connections 2024-03-17T23:46:09.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.775+0000 7fe0b4400700 1 --2- 172.21.15.12:0/2869776393 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe094038ed0 0x7fe09403b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:09.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.775+0000 7fe0b4400700 1 --2- 172.21.15.12:0/2869776393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0ac07dee0 0x7fe0ac19e180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:09.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.775+0000 7fe0b4400700 1 -- 172.21.15.12:0/2869776393 >> 172.21.15.12:0/2869776393 conn(0x7fe0ac103260 msgr2=0x7fe0ac103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:09.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.775+0000 7fe0b4400700 1 -- 172.21.15.12:0/2869776393 shutdown_connections 2024-03-17T23:46:09.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:08.776+0000 7fe0b4400700 1 -- 172.21.15.12:0/2869776393 wait complete. 2024-03-17T23:46:09.609 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling cephadm module... 2024-03-17T23:46:10.914 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:10 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: ignoring --setuser ceph since I am not root 2024-03-17T23:46:10.914 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:10 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: ignoring --setgroup ceph since I am not root 2024-03-17T23:46:10.914 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:10 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3464369777' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-17T23:46:11.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.211+0000 7f8c40e00700 1 Processor -- start 2024-03-17T23:46:11.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.211+0000 7f8c40e00700 1 -- start start 2024-03-17T23:46:11.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.212+0000 7f8c40e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 0x7f8c3c10a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:11.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.212+0000 7f8c40e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3c10a7d0 con 0x7f8c3c109e70 2024-03-17T23:46:11.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.212+0000 7f8c39e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 0x7f8c3c10a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:11.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.212+0000 7f8c39e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 0x7f8c3c10a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55006/0 (socket says 172.21.15.12:55006) 2024-03-17T23:46:11.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.212+0000 7f8c39e00700 1 -- 172.21.15.12:0/3919981882 learned_addr learned my addr 172.21.15.12:0/3919981882 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:11.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.213+0000 7f8c39e00700 1 -- 172.21.15.12:0/3919981882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c3c10afd0 con 0x7f8c3c109e70 2024-03-17T23:46:11.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.213+0000 7f8c39e00700 1 --2- 172.21.15.12:0/3919981882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 0x7f8c3c10a290 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c0098a0 tx=0x7f8c2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cefb426967a093cc server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:11.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.214+0000 7f8c39400700 1 -- 172.21.15.12:0/3919981882 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c2c004160 con 0x7f8c3c109e70 2024-03-17T23:46:11.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.214+0000 7f8c39400700 1 -- 172.21.15.12:0/3919981882 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8c2c004300 con 0x7f8c3c109e70 2024-03-17T23:46:11.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.214+0000 7f8c39400700 1 -- 172.21.15.12:0/3919981882 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c2c011b50 con 0x7f8c3c109e70 2024-03-17T23:46:11.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.214+0000 7f8c40e00700 1 -- 172.21.15.12:0/3919981882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 msgr2=0x7f8c3c10a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:11.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.214+0000 7f8c40e00700 1 --2- 172.21.15.12:0/3919981882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 0x7f8c3c10a290 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c0098a0 tx=0x7f8c2c009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:11.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.215+0000 7f8c40e00700 1 -- 172.21.15.12:0/3919981882 shutdown_connections 2024-03-17T23:46:11.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.215+0000 7f8c40e00700 1 --2- 172.21.15.12:0/3919981882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 0x7f8c3c10a290 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:11.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.215+0000 7f8c40e00700 1 -- 172.21.15.12:0/3919981882 >> 172.21.15.12:0/3919981882 conn(0x7f8c3c07d230 msgr2=0x7f8c3c07d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:11.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.215+0000 7f8c40e00700 1 -- 172.21.15.12:0/3919981882 shutdown_connections 2024-03-17T23:46:11.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.215+0000 7f8c40e00700 1 -- 172.21.15.12:0/3919981882 wait complete. 2024-03-17T23:46:11.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.216+0000 7f8c40e00700 1 Processor -- start 2024-03-17T23:46:11.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.216+0000 7f8c40e00700 1 -- start start 2024-03-17T23:46:11.216 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.216+0000 7f8c40e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 0x7f8c3c19e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:11.216 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.216+0000 7f8c40e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3c10e410 con 0x7f8c3c109e70 2024-03-17T23:46:11.216 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.217+0000 7f8c39e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 0x7f8c3c19e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:11.216 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.217+0000 7f8c39e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 0x7f8c3c19e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55018/0 (socket says 172.21.15.12:55018) 2024-03-17T23:46:11.216 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.217+0000 7f8c39e00700 1 -- 172.21.15.12:0/3464369777 learned_addr learned my addr 172.21.15.12:0/3464369777 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:11.216 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.217+0000 7f8c39e00700 1 -- 172.21.15.12:0/3464369777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c3c19e640 con 0x7f8c3c109e70 2024-03-17T23:46:11.216 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.217+0000 7f8c39e00700 1 --2- 172.21.15.12:0/3464369777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 0x7f8c3c19e100 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c011b60 tx=0x7f8c2c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:11.216 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.218+0000 7f8c32a00700 1 -- 172.21.15.12:0/3464369777 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c2c003b20 con 0x7f8c3c109e70 2024-03-17T23:46:11.216 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.218+0000 7f8c32a00700 1 -- 172.21.15.12:0/3464369777 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8c2c003cc0 con 0x7f8c3c109e70 2024-03-17T23:46:11.216 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.218+0000 7f8c40e00700 1 -- 172.21.15.12:0/3464369777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c3c19e8d0 con 0x7f8c3c109e70 2024-03-17T23:46:11.216 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.218+0000 7f8c32a00700 1 -- 172.21.15.12:0/3464369777 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c2c024840 con 0x7f8c3c109e70 2024-03-17T23:46:11.217 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.218+0000 7f8c40e00700 1 -- 172.21.15.12:0/3464369777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c3c1a15c0 con 0x7f8c3c109e70 2024-03-17T23:46:11.217 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.219+0000 7f8c40e00700 1 -- 172.21.15.12:0/3464369777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c3c063900 con 0x7f8c3c109e70 2024-03-17T23:46:11.217 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.219+0000 7f8c32a00700 1 -- 172.21.15.12:0/3464369777 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f8c2c022020 con 0x7f8c3c109e70 2024-03-17T23:46:11.217 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.220+0000 7f8c32a00700 1 --2- 172.21.15.12:0/3464369777 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f8c20038f20 0x7f8c2003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:11.217 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.220+0000 7f8c32a00700 1 -- 172.21.15.12:0/3464369777 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8c2c011e00 con 0x7f8c3c109e70 2024-03-17T23:46:11.217 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.220+0000 7f8c33e00700 1 --2- 172.21.15.12:0/3464369777 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f8c20038f20 0x7f8c2003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:11.217 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.221+0000 7f8c33e00700 1 --2- 172.21.15.12:0/3464369777 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f8c20038f20 0x7f8c2003b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8c24006fd0 tx=0x7f8c24006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:11.217 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.226+0000 7f8c32a00700 1 -- 172.21.15.12:0/3464369777 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8c2c00b3e0 con 0x7f8c3c109e70 2024-03-17T23:46:11.217 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.510+0000 7f8c40e00700 1 -- 172.21.15.12:0/3464369777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f8c3c02db70 con 0x7f8c3c109e70 2024-03-17T23:46:11.217 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.564+0000 7f8c32a00700 1 -- 172.21.15.12:0/3464369777 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f8c2c016260 con 0x7f8c3c109e70 2024-03-17T23:46:11.218 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.568+0000 7f8c32a00700 1 -- 172.21.15.12:0/3464369777 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f8c2c02c350 con 0x7f8c3c109e70 2024-03-17T23:46:11.218 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.568+0000 7f8c40e00700 1 -- 172.21.15.12:0/3464369777 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f8c20038f20 msgr2=0x7f8c2003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:11.218 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.568+0000 7f8c40e00700 1 --2- 172.21.15.12:0/3464369777 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f8c20038f20 0x7f8c2003b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8c24006fd0 tx=0x7f8c24006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:11.218 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.568+0000 7f8c40e00700 1 -- 172.21.15.12:0/3464369777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 msgr2=0x7f8c3c19e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:11.218 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.568+0000 7f8c40e00700 1 --2- 172.21.15.12:0/3464369777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 0x7f8c3c19e100 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c011b60 tx=0x7f8c2c011c40 comp rx=0 tx=0).stop 2024-03-17T23:46:11.218 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.569+0000 7f8c40e00700 1 -- 172.21.15.12:0/3464369777 shutdown_connections 2024-03-17T23:46:11.218 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.569+0000 7f8c40e00700 1 --2- 172.21.15.12:0/3464369777 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f8c20038f20 0x7f8c2003b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:11.218 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.569+0000 7f8c40e00700 1 --2- 172.21.15.12:0/3464369777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c3c109e70 0x7f8c3c19e100 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:11.218 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.569+0000 7f8c40e00700 1 -- 172.21.15.12:0/3464369777 >> 172.21.15.12:0/3464369777 conn(0x7f8c3c07d230 msgr2=0x7f8c3c107650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:11.218 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.570+0000 7f8c40e00700 1 -- 172.21.15.12:0/3464369777 shutdown_connections 2024-03-17T23:46:11.219 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:10.570+0000 7f8c40e00700 1 -- 172.21.15.12:0/3464369777 wait complete. 2024-03-17T23:46:11.489 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:11 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:11.362+0000 7fbfc4ea9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-17T23:46:11.794 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:11 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:11.490+0000 7fbfc4ea9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-17T23:46:11.795 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:11 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3464369777' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-17T23:46:11.795 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:11 smithi012 ceph-mon[31541]: mgrmap e5: x(active, since 5s) 2024-03-17T23:46:12.418 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:12 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:12.333+0000 7fbfc4ea9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-17T23:46:12.780 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:12 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/535906458' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-17T23:46:12.780 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:12 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:12.466+0000 7fbfc4ea9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-17T23:46:12.780 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:12 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:12.649+0000 7fbfc4ea9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-17T23:46:12.780 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:12 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:12.778+0000 7fbfc4ea9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-17T23:46:12.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-17T23:46:12.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-17T23:46:12.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-17T23:46:12.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-17T23:46:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-17T23:46:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-17T23:46:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.793+0000 7f9125400700 1 Processor -- start 2024-03-17T23:46:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.793+0000 7f9125400700 1 -- start start 2024-03-17T23:46:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.794+0000 7f9125400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 0x7f9120108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.794+0000 7f9125400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9120108640 con 0x7f9120107c50 2024-03-17T23:46:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.794+0000 7f911e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 0x7f9120108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.794+0000 7f911e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 0x7f9120108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55042/0 (socket says 172.21.15.12:55042) 2024-03-17T23:46:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.794+0000 7f911e800700 1 -- 172.21.15.12:0/2069452457 learned_addr learned my addr 172.21.15.12:0/2069452457 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.795+0000 7f911e800700 1 -- 172.21.15.12:0/2069452457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9120108e50 con 0x7f9120107c50 2024-03-17T23:46:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.795+0000 7f911e800700 1 --2- 172.21.15.12:0/2069452457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 0x7f9120108070 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9110009e80 tx=0x7f911000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51bc6ec38d276973 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.795+0000 7f911d400700 1 -- 172.21.15.12:0/2069452457 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9110017070 con 0x7f9120107c50 2024-03-17T23:46:12.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.795+0000 7f911d400700 1 -- 172.21.15.12:0/2069452457 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f911000b8c0 con 0x7f9120107c50 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.795+0000 7f911d400700 1 -- 172.21.15.12:0/2069452457 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91100045c0 con 0x7f9120107c50 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.796+0000 7f9125400700 1 -- 172.21.15.12:0/2069452457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 msgr2=0x7f9120108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.796+0000 7f9125400700 1 --2- 172.21.15.12:0/2069452457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 0x7f9120108070 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9110009e80 tx=0x7f911000b330 comp rx=0 tx=0).stop 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.796+0000 7f9125400700 1 -- 172.21.15.12:0/2069452457 shutdown_connections 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.796+0000 7f9125400700 1 --2- 172.21.15.12:0/2069452457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 0x7f9120108070 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.796+0000 7f9125400700 1 -- 172.21.15.12:0/2069452457 >> 172.21.15.12:0/2069452457 conn(0x7f9120103210 msgr2=0x7f9120105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.797+0000 7f9125400700 1 -- 172.21.15.12:0/2069452457 shutdown_connections 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.797+0000 7f9125400700 1 -- 172.21.15.12:0/2069452457 wait complete. 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.798+0000 7f9125400700 1 Processor -- start 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.798+0000 7f9125400700 1 -- start start 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.798+0000 7f9125400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 0x7f912019bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.798+0000 7f9125400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f912010c200 con 0x7f9120107c50 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.798+0000 7f911e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 0x7f912019bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:12.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.798+0000 7f911e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 0x7f912019bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55048/0 (socket says 172.21.15.12:55048) 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.798+0000 7f911e800700 1 -- 172.21.15.12:0/535906458 learned_addr learned my addr 172.21.15.12:0/535906458 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.799+0000 7f911e800700 1 -- 172.21.15.12:0/535906458 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f912019c470 con 0x7f9120107c50 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.799+0000 7f911e800700 1 --2- 172.21.15.12:0/535906458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 0x7f912019bf30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9110009530 tx=0x7f911000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.799+0000 7f9117400700 1 -- 172.21.15.12:0/535906458 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9110020020 con 0x7f9120107c50 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.799+0000 7f9117400700 1 -- 172.21.15.12:0/535906458 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9110003b50 con 0x7f9120107c50 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.799+0000 7f9117400700 1 -- 172.21.15.12:0/535906458 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9110017040 con 0x7f9120107c50 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.799+0000 7f9125400700 1 -- 172.21.15.12:0/535906458 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f912019c700 con 0x7f9120107c50 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.800+0000 7f9125400700 1 -- 172.21.15.12:0/535906458 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f912019f3a0 con 0x7f9120107c50 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.801+0000 7f9125400700 1 -- 172.21.15.12:0/535906458 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9120063900 con 0x7f9120107c50 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.801+0000 7f9117400700 1 -- 172.21.15.12:0/535906458 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f9110023020 con 0x7f9120107c50 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.802+0000 7f9117400700 1 --2- 172.21.15.12:0/535906458 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f9104038f70 0x7f910403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.802+0000 7f9117400700 1 -- 172.21.15.12:0/535906458 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f911000f050 con 0x7f9120107c50 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.802+0000 7f911de00700 1 -- 172.21.15.12:0/535906458 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f9104038f70 msgr2=0x7f910403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1563339918 2024-03-17T23:46:12.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.802+0000 7f911de00700 1 --2- 172.21.15.12:0/535906458 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f9104038f70 0x7f910403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:11.807+0000 7f9117400700 1 -- 172.21.15.12:0/535906458 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f911002a020 con 0x7f9120107c50 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.003+0000 7f911de00700 1 -- 172.21.15.12:0/535906458 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f9104038f70 msgr2=0x7f910403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1563339918 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.003+0000 7f911de00700 1 --2- 172.21.15.12:0/535906458 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f9104038f70 0x7f910403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.093+0000 7f9125400700 1 -- 172.21.15.12:0/535906458 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f912002db70 con 0x7f9120107c50 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.094+0000 7f9117400700 1 -- 172.21.15.12:0/535906458 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f911002d350 con 0x7f9120107c50 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.097+0000 7f9125400700 1 -- 172.21.15.12:0/535906458 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f9104038f70 msgr2=0x7f910403b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.097+0000 7f9125400700 1 --2- 172.21.15.12:0/535906458 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f9104038f70 0x7f910403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.097+0000 7f9125400700 1 -- 172.21.15.12:0/535906458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 msgr2=0x7f912019bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.097+0000 7f9125400700 1 --2- 172.21.15.12:0/535906458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 0x7f912019bf30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9110009530 tx=0x7f911000bd50 comp rx=0 tx=0).stop 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.097+0000 7f9125400700 1 -- 172.21.15.12:0/535906458 shutdown_connections 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.097+0000 7f9125400700 1 --2- 172.21.15.12:0/535906458 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7f9104038f70 0x7f910403b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.097+0000 7f9125400700 1 --2- 172.21.15.12:0/535906458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9120107c50 0x7f912019bf30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.097+0000 7f9125400700 1 -- 172.21.15.12:0/535906458 >> 172.21.15.12:0/535906458 conn(0x7f9120103210 msgr2=0x7f9120103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:12.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.097+0000 7f9125400700 1 -- 172.21.15.12:0/535906458 shutdown_connections 2024-03-17T23:46:12.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:12.097+0000 7f9125400700 1 -- 172.21.15.12:0/535906458 wait complete. 2024-03-17T23:46:12.801 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for the mgr to restart... 2024-03-17T23:46:12.801 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr epoch 5... 2024-03-17T23:46:13.085 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:12 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:12.909+0000 7fbfc4ea9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-17T23:46:13.370 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:13 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:13.259+0000 7fbfc4ea9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-17T23:46:13.620 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:13 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:13.409+0000 7fbfc4ea9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-17T23:46:13.620 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:13 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:13.525+0000 7fbfc4ea9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-17T23:46:13.882 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:13 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:13.657+0000 7fbfc4ea9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-17T23:46:13.882 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:13 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:13.881+0000 7fbfc4ea9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-17T23:46:15.668 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:15 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:15.202+0000 7fbfc4ea9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-17T23:46:15.669 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:15 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:15.390+0000 7fbfc4ea9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-17T23:46:16.168 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:15 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:15.819+0000 7fbfc4ea9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-17T23:46:16.168 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:15 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:15.942+0000 7fbfc4ea9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-17T23:46:16.461 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:16 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:16.207+0000 7fbfc4ea9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:46:16.850 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:16 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:16.460+0000 7fbfc4ea9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-17T23:46:17.169 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:16 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:16.849+0000 7fbfc4ea9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:46:17.492 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:17 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:17.173+0000 7fbfc4ea9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-17T23:46:17.918 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:17 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:17.491+0000 7fbfc4ea9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-17T23:46:18.918 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:18 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:18.558+0000 7fbfc4ea9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-17T23:46:19.627 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:19.279+0000 7fbfc4ea9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-17T23:46:19.627 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:19.413+0000 7fbfc4ea9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-17T23:46:19.918 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:19.703+0000 7fbfc4ea9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-17T23:46:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-mon[31541]: Active manager daemon x restarted 2024-03-17T23:46:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-mon[31541]: Activating manager daemon x 2024-03-17T23:46:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-mon[31541]: osdmap e2: 0 total, 0 up, 0 in 2024-03-17T23:46:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-mon[31541]: mgrmap e6: x(active, starting, since 0.00879992s) 2024-03-17T23:46:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:46:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-17T23:46:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-17T23:46:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-17T23:46:19.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-17T23:46:19.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-mon[31541]: Manager daemon x is now available 2024-03-17T23:46:19.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:19 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' 2024-03-17T23:46:21.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:20 smithi012 ceph-mon[31541]: Found migration_current of "None". Setting to last migration. 2024-03-17T23:46:21.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:20 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' 2024-03-17T23:46:21.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:20 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:46:21.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:20 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:46:21.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:20 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:46:21.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:20 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-17T23:46:21.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:20 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-17T23:46:21.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:20 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' 2024-03-17T23:46:21.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:20 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' 2024-03-17T23:46:21.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:20 smithi012 ceph-mon[31541]: [17/Mar/2024:23:46:20] ENGINE Bus STARTING 2024-03-17T23:46:21.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:20 smithi012 ceph-mon[31541]: mgrmap e7: x(active, since 1.00865s) 2024-03-17T23:46:21.445 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-17T23:46:21.445 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-17T23:46:21.445 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-17T23:46:21.445 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-17T23:46:21.445 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.410+0000 7fe947600700 1 Processor -- start 2024-03-17T23:46:21.445 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.410+0000 7fe947600700 1 -- start start 2024-03-17T23:46:21.445 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.410+0000 7fe947600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 0x7fe940108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:21.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.410+0000 7fe947600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe940108640 con 0x7fe940107c50 2024-03-17T23:46:21.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.411+0000 7fe944c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 0x7fe940108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:21.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.411+0000 7fe944c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 0x7fe940108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55062/0 (socket says 172.21.15.12:55062) 2024-03-17T23:46:21.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.411+0000 7fe944c00700 1 -- 172.21.15.12:0/1650224156 learned_addr learned my addr 172.21.15.12:0/1650224156 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:21.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.411+0000 7fe944c00700 1 -- 172.21.15.12:0/1650224156 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe940108e50 con 0x7fe940107c50 2024-03-17T23:46:21.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.412+0000 7fe944c00700 1 --2- 172.21.15.12:0/1650224156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 0x7fe940108070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe9340098a0 tx=0x7fe934009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2bea141b7dad3f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:21.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.412+0000 7fe93f400700 1 -- 172.21.15.12:0/1650224156 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe934004160 con 0x7fe940107c50 2024-03-17T23:46:21.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.412+0000 7fe93f400700 1 -- 172.21.15.12:0/1650224156 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe934004300 con 0x7fe940107c50 2024-03-17T23:46:21.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.412+0000 7fe93f400700 1 -- 172.21.15.12:0/1650224156 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe934011b50 con 0x7fe940107c50 2024-03-17T23:46:21.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.412+0000 7fe947600700 1 -- 172.21.15.12:0/1650224156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 msgr2=0x7fe940108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:21.446 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.412+0000 7fe947600700 1 --2- 172.21.15.12:0/1650224156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 0x7fe940108070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe9340098a0 tx=0x7fe934009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:21.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.413+0000 7fe947600700 1 -- 172.21.15.12:0/1650224156 shutdown_connections 2024-03-17T23:46:21.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.413+0000 7fe947600700 1 --2- 172.21.15.12:0/1650224156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 0x7fe940108070 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:21.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.413+0000 7fe947600700 1 -- 172.21.15.12:0/1650224156 >> 172.21.15.12:0/1650224156 conn(0x7fe940103210 msgr2=0x7fe940105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:21.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.413+0000 7fe947600700 1 -- 172.21.15.12:0/1650224156 shutdown_connections 2024-03-17T23:46:21.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.413+0000 7fe947600700 1 -- 172.21.15.12:0/1650224156 wait complete. 2024-03-17T23:46:21.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.414+0000 7fe947600700 1 Processor -- start 2024-03-17T23:46:21.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.414+0000 7fe947600700 1 -- start start 2024-03-17T23:46:21.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.414+0000 7fe947600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 0x7fe94019e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:21.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.414+0000 7fe947600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe94010c200 con 0x7fe940107c50 2024-03-17T23:46:21.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.415+0000 7fe944c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 0x7fe94019e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:21.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.415+0000 7fe944c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 0x7fe94019e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55072/0 (socket says 172.21.15.12:55072) 2024-03-17T23:46:21.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.415+0000 7fe944c00700 1 -- 172.21.15.12:0/4178488724 learned_addr learned my addr 172.21.15.12:0/4178488724 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:21.447 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.415+0000 7fe944c00700 1 -- 172.21.15.12:0/4178488724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe94019e6b0 con 0x7fe940107c50 2024-03-17T23:46:21.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.415+0000 7fe944c00700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 0x7fe94019e170 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe934000c00 tx=0x7fe934011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:21.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.416+0000 7fe93d600700 1 -- 172.21.15.12:0/4178488724 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe934003b20 con 0x7fe940107c50 2024-03-17T23:46:21.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.416+0000 7fe93d600700 1 -- 172.21.15.12:0/4178488724 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe934003cc0 con 0x7fe940107c50 2024-03-17T23:46:21.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.416+0000 7fe93d600700 1 -- 172.21.15.12:0/4178488724 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe934024910 con 0x7fe940107c50 2024-03-17T23:46:21.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.416+0000 7fe947600700 1 -- 172.21.15.12:0/4178488724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe94019e940 con 0x7fe940107c50 2024-03-17T23:46:21.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.416+0000 7fe947600700 1 -- 172.21.15.12:0/4178488724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9401a1630 con 0x7fe940107c50 2024-03-17T23:46:21.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.417+0000 7fe93d600700 1 -- 172.21.15.12:0/4178488724 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fe934022020 con 0x7fe940107c50 2024-03-17T23:46:21.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.417+0000 7fe93d600700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 0x7fe92803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:21.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.417+0000 7fe93d600700 1 -- 172.21.15.12:0/4178488724 --> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe92803bb40 con 0x7fe928038f70 2024-03-17T23:46:21.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.417+0000 7fe93d600700 1 -- 172.21.15.12:0/4178488724 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe934011e00 con 0x7fe940107c50 2024-03-17T23:46:21.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.417+0000 7fe93fe00700 1 -- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 msgr2=0x7fe92803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1563339918 2024-03-17T23:46:21.448 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.418+0000 7fe93fe00700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 0x7fe92803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-17T23:46:21.449 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.619+0000 7fe93fe00700 1 -- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 msgr2=0x7fe92803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1563339918 2024-03-17T23:46:21.449 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:13.619+0000 7fe93fe00700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 0x7fe92803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-17T23:46:21.449 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:14.020+0000 7fe93fe00700 1 -- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 msgr2=0x7fe92803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1563339918 2024-03-17T23:46:21.449 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:14.020+0000 7fe93fe00700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 0x7fe92803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-17T23:46:21.449 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:14.821+0000 7fe93fe00700 1 -- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 msgr2=0x7fe92803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1563339918 2024-03-17T23:46:21.449 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:14.821+0000 7fe93fe00700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 0x7fe92803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-17T23:46:21.449 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:16.423+0000 7fe93fe00700 1 -- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 msgr2=0x7fe92803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1563339918 2024-03-17T23:46:21.449 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:16.423+0000 7fe93fe00700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 0x7fe92803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-17T23:46:21.449 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:19.626+0000 7fe93fe00700 1 -- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 msgr2=0x7fe92803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1563339918 2024-03-17T23:46:21.449 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:19.626+0000 7fe93fe00700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 0x7fe92803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-17T23:46:21.449 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:19.712+0000 7fe93d600700 1 -- 172.21.15.12:0/4178488724 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7fe934021b40 con 0x7fe940107c50 2024-03-17T23:46:21.449 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:19.712+0000 7fe93d600700 1 -- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 msgr2=0x7fe92803b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-17T23:46:21.449 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:19.712+0000 7fe93d600700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 0x7fe92803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:21.450 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.711+0000 7fe93d600700 1 -- 172.21.15.12:0/4178488724 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7fe93400b770 con 0x7fe940107c50 2024-03-17T23:46:21.450 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.712+0000 7fe93d600700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fe92803c860 0x7fe92803ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:21.450 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.712+0000 7fe93d600700 1 -- 172.21.15.12:0/4178488724 --> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe93400af30 con 0x7fe92803c860 2024-03-17T23:46:21.450 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.713+0000 7fe93fe00700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fe92803c860 0x7fe92803ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:21.450 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.715+0000 7fe93fe00700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fe92803c860 0x7fe92803ec50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe92c003a10 tx=0x7fe92c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:21.450 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.715+0000 7fe93d600700 1 -- 172.21.15.12:0/4178488724 <== mgr.14120 v2:172.21.15.12:6800/2526360059 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fe93400af30 con 0x7fe92803c860 2024-03-17T23:46:21.450 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.723+0000 7fe947600700 1 -- 172.21.15.12:0/4178488724 --> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fe940063900 con 0x7fe92803c860 2024-03-17T23:46:21.450 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.723+0000 7fe93d600700 1 -- 172.21.15.12:0/4178488724 <== mgr.14120 v2:172.21.15.12:6800/2526360059 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fe940063900 con 0x7fe92803c860 2024-03-17T23:46:21.450 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.724+0000 7fe947600700 1 -- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fe92803c860 msgr2=0x7fe92803ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:21.450 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.724+0000 7fe947600700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fe92803c860 0x7fe92803ec50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe92c003a10 tx=0x7fe92c0092b0 comp rx=0 tx=0).stop 2024-03-17T23:46:21.450 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.724+0000 7fe947600700 1 -- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 msgr2=0x7fe94019e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:21.450 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.724+0000 7fe947600700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 0x7fe94019e170 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe934000c00 tx=0x7fe934011c10 comp rx=0 tx=0).stop 2024-03-17T23:46:21.451 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.724+0000 7fe947600700 1 -- 172.21.15.12:0/4178488724 shutdown_connections 2024-03-17T23:46:21.451 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.724+0000 7fe947600700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fe92803c860 0x7fe92803ec50 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:21.451 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.724+0000 7fe947600700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe940107c50 0x7fe94019e170 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:21.451 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.724+0000 7fe947600700 1 --2- 172.21.15.12:0/4178488724 >> [v2:172.21.15.12:6800/1563339918,v1:172.21.15.12:6801/1563339918] conn(0x7fe928038f70 0x7fe92803b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:21.451 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.724+0000 7fe947600700 1 -- 172.21.15.12:0/4178488724 >> 172.21.15.12:0/4178488724 conn(0x7fe940103210 msgr2=0x7fe940103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:21.451 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.724+0000 7fe947600700 1 -- 172.21.15.12:0/4178488724 shutdown_connections 2024-03-17T23:46:21.451 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:20.724+0000 7fe947600700 1 -- 172.21.15.12:0/4178488724 wait complete. 2024-03-17T23:46:21.451 INFO:teuthology.orchestra.run.smithi012.stdout:mgr epoch 5 is available 2024-03-17T23:46:21.451 INFO:teuthology.orchestra.run.smithi012.stdout:Setting orchestrator backend to cephadm... 2024-03-17T23:46:22.041 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:21 smithi012 ceph-mon[31541]: [17/Mar/2024:23:46:20] ENGINE Serving on https://172.21.15.12:7150 2024-03-17T23:46:22.041 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:21 smithi012 ceph-mon[31541]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-17T23:46:22.041 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:21 smithi012 ceph-mon[31541]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-17T23:46:22.041 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:21 smithi012 ceph-mon[31541]: [17/Mar/2024:23:46:20] ENGINE Serving on http://172.21.15.12:8765 2024-03-17T23:46:22.041 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:21 smithi012 ceph-mon[31541]: [17/Mar/2024:23:46:20] ENGINE Bus STARTED 2024-03-17T23:46:22.041 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:21 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:46:22.989 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:22 smithi012 ceph-mon[31541]: mgrmap e8: x(active, since 2s) 2024-03-17T23:46:22.989 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:22 smithi012 ceph-mon[31541]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:46:22.989 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:22 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' 2024-03-17T23:46:22.989 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:22 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:46:23.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.039+0000 7f6e5e800700 1 Processor -- start 2024-03-17T23:46:23.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.040+0000 7f6e5e800700 1 -- start start 2024-03-17T23:46:23.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.040+0000 7f6e5e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 0x7f6e581110c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:23.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.040+0000 7f6e5e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6e580824d0 con 0x7f6e5810ecd0 2024-03-17T23:46:23.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.040+0000 7f6e57400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 0x7f6e581110c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:23.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.041+0000 7f6e57400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 0x7f6e581110c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47446/0 (socket says 172.21.15.12:47446) 2024-03-17T23:46:23.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.041+0000 7f6e57400700 1 -- 172.21.15.12:0/1255585441 learned_addr learned my addr 172.21.15.12:0/1255585441 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:23.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.041+0000 7f6e57400700 1 -- 172.21.15.12:0/1255585441 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e58111600 con 0x7f6e5810ecd0 2024-03-17T23:46:23.001 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.042+0000 7f6e57400700 1 --2- 172.21.15.12:0/1255585441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 0x7f6e581110c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6e480098a0 tx=0x7f6e48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e5ee1d274fcb136 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.042+0000 7f6e56000700 1 -- 172.21.15.12:0/1255585441 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6e48004160 con 0x7f6e5810ecd0 2024-03-17T23:46:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.042+0000 7f6e56000700 1 -- 172.21.15.12:0/1255585441 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6e48004300 con 0x7f6e5810ecd0 2024-03-17T23:46:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.042+0000 7f6e56000700 1 -- 172.21.15.12:0/1255585441 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6e48011b50 con 0x7f6e5810ecd0 2024-03-17T23:46:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.043+0000 7f6e5e800700 1 -- 172.21.15.12:0/1255585441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 msgr2=0x7f6e581110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.043+0000 7f6e5e800700 1 --2- 172.21.15.12:0/1255585441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 0x7f6e581110c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6e480098a0 tx=0x7f6e48009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.043+0000 7f6e5e800700 1 -- 172.21.15.12:0/1255585441 shutdown_connections 2024-03-17T23:46:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.043+0000 7f6e5e800700 1 --2- 172.21.15.12:0/1255585441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 0x7f6e581110c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.043+0000 7f6e5e800700 1 -- 172.21.15.12:0/1255585441 >> 172.21.15.12:0/1255585441 conn(0x7f6e5807d230 msgr2=0x7f6e5807f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.043+0000 7f6e5e800700 1 -- 172.21.15.12:0/1255585441 shutdown_connections 2024-03-17T23:46:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.043+0000 7f6e5e800700 1 -- 172.21.15.12:0/1255585441 wait complete. 2024-03-17T23:46:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.044+0000 7f6e5e800700 1 Processor -- start 2024-03-17T23:46:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.044+0000 7f6e5e800700 1 -- start start 2024-03-17T23:46:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.045+0000 7f6e5e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 0x7f6e581a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.045+0000 7f6e5e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6e58112630 con 0x7f6e5810ecd0 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.045+0000 7f6e57400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 0x7f6e581a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.045+0000 7f6e57400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 0x7f6e581a24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47462/0 (socket says 172.21.15.12:47462) 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.045+0000 7f6e57400700 1 -- 172.21.15.12:0/288073479 learned_addr learned my addr 172.21.15.12:0/288073479 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.045+0000 7f6e57400700 1 -- 172.21.15.12:0/288073479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e581a2a20 con 0x7f6e5810ecd0 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.046+0000 7f6e57400700 1 --2- 172.21.15.12:0/288073479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 0x7f6e581a24e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e48011670 tx=0x7f6e48011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.046+0000 7f6e4fe00700 1 -- 172.21.15.12:0/288073479 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6e480039c0 con 0x7f6e5810ecd0 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.046+0000 7f6e4fe00700 1 -- 172.21.15.12:0/288073479 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6e48011e50 con 0x7f6e5810ecd0 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.046+0000 7f6e5e800700 1 -- 172.21.15.12:0/288073479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e581a2cb0 con 0x7f6e5810ecd0 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.046+0000 7f6e4fe00700 1 -- 172.21.15.12:0/288073479 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6e4801a5c0 con 0x7f6e5810ecd0 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.046+0000 7f6e5e800700 1 -- 172.21.15.12:0/288073479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e581a59a0 con 0x7f6e5810ecd0 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.048+0000 7f6e4fe00700 1 -- 172.21.15.12:0/288073479 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f6e48024430 con 0x7f6e5810ecd0 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.048+0000 7f6e5e800700 1 -- 172.21.15.12:0/288073479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e58063900 con 0x7f6e5810ecd0 2024-03-17T23:46:23.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.048+0000 7f6e4fe00700 1 --2- 172.21.15.12:0/288073479 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f6e3c038ed0 0x7f6e3c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:23.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.048+0000 7f6e4fe00700 1 -- 172.21.15.12:0/288073479 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6e4804d320 con 0x7f6e5810ecd0 2024-03-17T23:46:23.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.048+0000 7f6e56a00700 1 --2- 172.21.15.12:0/288073479 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f6e3c038ed0 0x7f6e3c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:23.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.049+0000 7f6e56a00700 1 --2- 172.21.15.12:0/288073479 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f6e3c038ed0 0x7f6e3c03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6e44006fd0 tx=0x7f6e44006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:23.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.055+0000 7f6e4fe00700 1 -- 172.21.15.12:0/288073479 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6e48017ca0 con 0x7f6e5810ecd0 2024-03-17T23:46:23.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.297+0000 7f6e5e800700 1 -- 172.21.15.12:0/288073479 --> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f6e5802dbe0 con 0x7f6e3c038ed0 2024-03-17T23:46:23.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.309+0000 7f6e4fe00700 1 -- 172.21.15.12:0/288073479 <== mgr.14120 v2:172.21.15.12:6800/2526360059 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6e5802dbe0 con 0x7f6e3c038ed0 2024-03-17T23:46:23.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.312+0000 7f6e5e800700 1 -- 172.21.15.12:0/288073479 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f6e3c038ed0 msgr2=0x7f6e3c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:23.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.312+0000 7f6e5e800700 1 --2- 172.21.15.12:0/288073479 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f6e3c038ed0 0x7f6e3c03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6e44006fd0 tx=0x7f6e44006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:23.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.312+0000 7f6e5e800700 1 -- 172.21.15.12:0/288073479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 msgr2=0x7f6e581a24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:23.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.312+0000 7f6e5e800700 1 --2- 172.21.15.12:0/288073479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 0x7f6e581a24e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e48011670 tx=0x7f6e48011b60 comp rx=0 tx=0).stop 2024-03-17T23:46:23.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.312+0000 7f6e5e800700 1 -- 172.21.15.12:0/288073479 shutdown_connections 2024-03-17T23:46:23.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.312+0000 7f6e5e800700 1 --2- 172.21.15.12:0/288073479 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f6e3c038ed0 0x7f6e3c03b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:23.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.312+0000 7f6e5e800700 1 --2- 172.21.15.12:0/288073479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e5810ecd0 0x7f6e581a24e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:23.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.312+0000 7f6e5e800700 1 -- 172.21.15.12:0/288073479 >> 172.21.15.12:0/288073479 conn(0x7f6e5807d230 msgr2=0x7f6e5807df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:23.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.313+0000 7f6e5e800700 1 -- 172.21.15.12:0/288073479 shutdown_connections 2024-03-17T23:46:23.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:22.313+0000 7f6e5e800700 1 -- 172.21.15.12:0/288073479 wait complete. 2024-03-17T23:46:24.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-17T23:46:24.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.603+0000 7f9990600700 1 Processor -- start 2024-03-17T23:46:24.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.603+0000 7f9990600700 1 -- start start 2024-03-17T23:46:24.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.603+0000 7f9990600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 0x7f9988106700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:24.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.603+0000 7f9990600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9988106cd0 con 0x7f99881062e0 2024-03-17T23:46:24.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.604+0000 7f998dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 0x7f9988106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:24.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.604+0000 7f998dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 0x7f9988106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47464/0 (socket says 172.21.15.12:47464) 2024-03-17T23:46:24.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.604+0000 7f998dc00700 1 -- 172.21.15.12:0/4087615241 learned_addr learned my addr 172.21.15.12:0/4087615241 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:24.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.604+0000 7f998dc00700 1 -- 172.21.15.12:0/4087615241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99881074b0 con 0x7f99881062e0 2024-03-17T23:46:24.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.605+0000 7f998dc00700 1 --2- 172.21.15.12:0/4087615241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 0x7f9988106700 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f997c0098a0 tx=0x7f997c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7f2f91b21e96cf server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:24.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.605+0000 7f9987e00700 1 -- 172.21.15.12:0/4087615241 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f997c004160 con 0x7f99881062e0 2024-03-17T23:46:24.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.605+0000 7f9987e00700 1 -- 172.21.15.12:0/4087615241 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f997c004300 con 0x7f99881062e0 2024-03-17T23:46:24.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.605+0000 7f9987e00700 1 -- 172.21.15.12:0/4087615241 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f997c011b50 con 0x7f99881062e0 2024-03-17T23:46:24.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.606+0000 7f9990600700 1 -- 172.21.15.12:0/4087615241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 msgr2=0x7f9988106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:24.540 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.606+0000 7f9990600700 1 --2- 172.21.15.12:0/4087615241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 0x7f9988106700 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f997c0098a0 tx=0x7f997c009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:24.540 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.606+0000 7f9990600700 1 -- 172.21.15.12:0/4087615241 shutdown_connections 2024-03-17T23:46:24.540 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.606+0000 7f9990600700 1 --2- 172.21.15.12:0/4087615241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 0x7f9988106700 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:24.540 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.606+0000 7f9990600700 1 -- 172.21.15.12:0/4087615241 >> 172.21.15.12:0/4087615241 conn(0x7f9988101420 msgr2=0x7f9988103840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:24.540 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.606+0000 7f9990600700 1 -- 172.21.15.12:0/4087615241 shutdown_connections 2024-03-17T23:46:24.540 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.606+0000 7f9990600700 1 -- 172.21.15.12:0/4087615241 wait complete. 2024-03-17T23:46:24.541 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.607+0000 7f9990600700 1 Processor -- start 2024-03-17T23:46:24.541 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.607+0000 7f9990600700 1 -- start start 2024-03-17T23:46:24.541 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.607+0000 7f9990600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 0x7f998819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:24.541 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.607+0000 7f9990600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f998810a860 con 0x7f99881062e0 2024-03-17T23:46:24.541 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.608+0000 7f998dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 0x7f998819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:24.541 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.608+0000 7f998dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 0x7f998819de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47466/0 (socket says 172.21.15.12:47466) 2024-03-17T23:46:24.541 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.608+0000 7f998dc00700 1 -- 172.21.15.12:0/4119555308 learned_addr learned my addr 172.21.15.12:0/4119555308 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:24.541 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.608+0000 7f998dc00700 1 -- 172.21.15.12:0/4119555308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f998819e380 con 0x7f99881062e0 2024-03-17T23:46:24.541 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.608+0000 7f998dc00700 1 --2- 172.21.15.12:0/4119555308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 0x7f998819de40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f997c000c00 tx=0x7f997c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:24.541 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.609+0000 7f9986000700 1 -- 172.21.15.12:0/4119555308 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f997c003b20 con 0x7f99881062e0 2024-03-17T23:46:24.541 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.609+0000 7f9986000700 1 -- 172.21.15.12:0/4119555308 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f997c003cc0 con 0x7f99881062e0 2024-03-17T23:46:24.541 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.609+0000 7f9990600700 1 -- 172.21.15.12:0/4119555308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f998819e610 con 0x7f99881062e0 2024-03-17T23:46:24.541 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.609+0000 7f9986000700 1 -- 172.21.15.12:0/4119555308 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f997c024910 con 0x7f99881062e0 2024-03-17T23:46:24.542 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.609+0000 7f9990600700 1 -- 172.21.15.12:0/4119555308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99881a1300 con 0x7f99881062e0 2024-03-17T23:46:24.542 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.610+0000 7f9986000700 1 -- 172.21.15.12:0/4119555308 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f997c022020 con 0x7f99881062e0 2024-03-17T23:46:24.542 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.610+0000 7f997b400700 1 -- 172.21.15.12:0/4119555308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9988063900 con 0x7f99881062e0 2024-03-17T23:46:24.542 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.611+0000 7f9986000700 1 --2- 172.21.15.12:0/4119555308 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f9970038f70 0x7f997003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:24.542 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.611+0000 7f9986000700 1 -- 172.21.15.12:0/4119555308 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f997c011e00 con 0x7f99881062e0 2024-03-17T23:46:24.542 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.611+0000 7f998d200700 1 --2- 172.21.15.12:0/4119555308 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f9970038f70 0x7f997003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:24.542 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.612+0000 7f998d200700 1 --2- 172.21.15.12:0/4119555308 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f9970038f70 0x7f997003b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9974006fd0 tx=0x7f9974006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:24.542 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.617+0000 7f9986000700 1 -- 172.21.15.12:0/4119555308 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f997c01acd0 con 0x7f99881062e0 2024-03-17T23:46:24.542 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.844+0000 7f997b400700 1 -- 172.21.15.12:0/4119555308 --> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f9988063070 con 0x7f9970038f70 2024-03-17T23:46:24.542 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.846+0000 7f9986000700 1 -- 172.21.15.12:0/4119555308 <== mgr.14120 v2:172.21.15.12:6800/2526360059 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f9988063070 con 0x7f9970038f70 2024-03-17T23:46:24.542 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.848+0000 7f997b400700 1 -- 172.21.15.12:0/4119555308 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f9970038f70 msgr2=0x7f997003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:24.542 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.848+0000 7f997b400700 1 --2- 172.21.15.12:0/4119555308 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f9970038f70 0x7f997003b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9974006fd0 tx=0x7f9974006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:24.542 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.848+0000 7f997b400700 1 -- 172.21.15.12:0/4119555308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 msgr2=0x7f998819de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:24.543 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.848+0000 7f997b400700 1 --2- 172.21.15.12:0/4119555308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 0x7f998819de40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f997c000c00 tx=0x7f997c011c10 comp rx=0 tx=0).stop 2024-03-17T23:46:24.543 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.848+0000 7f997b400700 1 -- 172.21.15.12:0/4119555308 shutdown_connections 2024-03-17T23:46:24.543 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.848+0000 7f997b400700 1 --2- 172.21.15.12:0/4119555308 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f9970038f70 0x7f997003b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:24.543 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.848+0000 7f997b400700 1 --2- 172.21.15.12:0/4119555308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99881062e0 0x7f998819de40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:24.543 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.848+0000 7f997b400700 1 -- 172.21.15.12:0/4119555308 >> 172.21.15.12:0/4119555308 conn(0x7f9988101420 msgr2=0x7f99881026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:24.543 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.848+0000 7f997b400700 1 -- 172.21.15.12:0/4119555308 shutdown_connections 2024-03-17T23:46:24.543 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:23.848+0000 7f997b400700 1 -- 172.21.15.12:0/4119555308 wait complete. 2024-03-17T23:46:24.543 INFO:teuthology.orchestra.run.smithi012.stdout:Generating ssh key... 2024-03-17T23:46:24.834 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:24 smithi012 ceph-mon[31541]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:46:26.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:25 smithi012 ceph-mon[31541]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:46:26.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:25 smithi012 ceph-mon[31541]: Generating ssh key... 2024-03-17T23:46:26.393 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: Generating public/private rsa key pair. 2024-03-17T23:46:26.393 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: Your identification has been saved in /tmp/tmpm3agz24u/key. 2024-03-17T23:46:26.393 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: Your public key has been saved in /tmp/tmpm3agz24u/key.pub. 2024-03-17T23:46:26.393 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: The key fingerprint is: 2024-03-17T23:46:26.393 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: SHA256:JFkIsGCI+ZDW8VELbsTKbavXXZxuu+nFW1V5WOC5zWk ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:46:26.393 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: The key's randomart image is: 2024-03-17T23:46:26.393 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: +---[RSA 3072]----+ 2024-03-17T23:46:26.393 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: |++oo+=.o. ...| 2024-03-17T23:46:26.393 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: |Bo o+.+o. . +.| 2024-03-17T23:46:26.393 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: |.oo o+o.. +.o| 2024-03-17T23:46:26.394 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: | .o.o o ++| 2024-03-17T23:46:26.394 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: | . . S . . .E+| 2024-03-17T23:46:26.394 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: | . +. . .| 2024-03-17T23:46:26.394 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: | . . . o o . | 2024-03-17T23:46:26.394 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: | . . . . oo o | 2024-03-17T23:46:26.394 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: | . o=o. | 2024-03-17T23:46:26.394 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: +----[SHA256]-----+ 2024-03-17T23:46:26.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.133+0000 7f8904400700 1 Processor -- start 2024-03-17T23:46:26.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.133+0000 7f8904400700 1 -- start start 2024-03-17T23:46:26.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.133+0000 7f8904400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 0x7f88fc106950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:26.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.133+0000 7f8904400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc0759c0 con 0x7f88fc106530 2024-03-17T23:46:26.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.134+0000 7f8901a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 0x7f88fc106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:26.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.134+0000 7f8901a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 0x7f88fc106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50424/0 (socket says 172.21.15.12:50424) 2024-03-17T23:46:26.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.134+0000 7f8901a00700 1 -- 172.21.15.12:0/3984304825 learned_addr learned my addr 172.21.15.12:0/3984304825 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:26.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.134+0000 7f8901a00700 1 -- 172.21.15.12:0/3984304825 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88fc106e90 con 0x7f88fc106530 2024-03-17T23:46:26.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.135+0000 7f8901a00700 1 --2- 172.21.15.12:0/3984304825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 0x7f88fc106950 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009e80 tx=0x7f88f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d71216566cbfe49 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:26.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.135+0000 7f88fbe00700 1 -- 172.21.15.12:0/3984304825 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88f0017070 con 0x7f88fc106530 2024-03-17T23:46:26.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.135+0000 7f88fbe00700 1 -- 172.21.15.12:0/3984304825 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f88f000b8c0 con 0x7f88fc106530 2024-03-17T23:46:26.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.135+0000 7f88fbe00700 1 -- 172.21.15.12:0/3984304825 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88f00045c0 con 0x7f88fc106530 2024-03-17T23:46:26.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.136+0000 7f8904400700 1 -- 172.21.15.12:0/3984304825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 msgr2=0x7f88fc106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:26.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.136+0000 7f8904400700 1 --2- 172.21.15.12:0/3984304825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 0x7f88fc106950 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009e80 tx=0x7f88f000b330 comp rx=0 tx=0).stop 2024-03-17T23:46:26.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.136+0000 7f8904400700 1 -- 172.21.15.12:0/3984304825 shutdown_connections 2024-03-17T23:46:26.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.136+0000 7f8904400700 1 --2- 172.21.15.12:0/3984304825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 0x7f88fc106950 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:26.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.136+0000 7f8904400700 1 -- 172.21.15.12:0/3984304825 >> 172.21.15.12:0/3984304825 conn(0x7f88fc102130 msgr2=0x7f88fc104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.136+0000 7f8904400700 1 -- 172.21.15.12:0/3984304825 shutdown_connections 2024-03-17T23:46:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.136+0000 7f8904400700 1 -- 172.21.15.12:0/3984304825 wait complete. 2024-03-17T23:46:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.137+0000 7f8904400700 1 Processor -- start 2024-03-17T23:46:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.137+0000 7f8904400700 1 -- start start 2024-03-17T23:46:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.137+0000 7f8904400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 0x7f88fc19ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.137+0000 7f8904400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc107f20 con 0x7f88fc106530 2024-03-17T23:46:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.138+0000 7f8901a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 0x7f88fc19ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.138+0000 7f8901a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 0x7f88fc19ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50432/0 (socket says 172.21.15.12:50432) 2024-03-17T23:46:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.138+0000 7f8901a00700 1 -- 172.21.15.12:0/1611760593 learned_addr learned my addr 172.21.15.12:0/1611760593 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.138+0000 7f8901a00700 1 -- 172.21.15.12:0/1611760593 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88fc19e2f0 con 0x7f88fc106530 2024-03-17T23:46:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.138+0000 7f8901a00700 1 --2- 172.21.15.12:0/1611760593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 0x7f88fc19ddb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f88f0004350 tx=0x7f88f000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.139+0000 7f88fa000700 1 -- 172.21.15.12:0/1611760593 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88f0020020 con 0x7f88fc106530 2024-03-17T23:46:26.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.139+0000 7f88fa000700 1 -- 172.21.15.12:0/1611760593 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f88f0003c10 con 0x7f88fc106530 2024-03-17T23:46:26.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.139+0000 7f8904400700 1 -- 172.21.15.12:0/1611760593 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88fc19e580 con 0x7f88fc106530 2024-03-17T23:46:26.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.139+0000 7f88fa000700 1 -- 172.21.15.12:0/1611760593 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88f0017040 con 0x7f88fc106530 2024-03-17T23:46:26.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.139+0000 7f8904400700 1 -- 172.21.15.12:0/1611760593 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88fc1a1270 con 0x7f88fc106530 2024-03-17T23:46:26.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.140+0000 7f8904400700 1 -- 172.21.15.12:0/1611760593 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88fc063900 con 0x7f88fc106530 2024-03-17T23:46:26.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.141+0000 7f88fa000700 1 -- 172.21.15.12:0/1611760593 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f88f0023020 con 0x7f88fc106530 2024-03-17T23:46:26.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.141+0000 7f88fa000700 1 --2- 172.21.15.12:0/1611760593 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f88e4038f20 0x7f88e403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:26.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.141+0000 7f88fa000700 1 -- 172.21.15.12:0/1611760593 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f88f000f050 con 0x7f88fc106530 2024-03-17T23:46:26.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.141+0000 7f8901000700 1 --2- 172.21.15.12:0/1611760593 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f88e4038f20 0x7f88e403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:26.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.142+0000 7f8901000700 1 --2- 172.21.15.12:0/1611760593 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f88e4038f20 0x7f88e403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f88e8006fd0 tx=0x7f88e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:26.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.146+0000 7f88fa000700 1 -- 172.21.15.12:0/1611760593 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f88f0053080 con 0x7f88fc106530 2024-03-17T23:46:26.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:25.371+0000 7f8904400700 1 -- 172.21.15.12:0/1611760593 --> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f88fc02df50 con 0x7f88e4038f20 2024-03-17T23:46:26.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:26.073+0000 7f88fa000700 1 -- 172.21.15.12:0/1611760593 <== mgr.14120 v2:172.21.15.12:6800/2526360059 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f88fc02df50 con 0x7f88e4038f20 2024-03-17T23:46:26.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:26.076+0000 7f8904400700 1 -- 172.21.15.12:0/1611760593 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f88e4038f20 msgr2=0x7f88e403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:26.741 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:26.076+0000 7f8904400700 1 --2- 172.21.15.12:0/1611760593 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f88e4038f20 0x7f88e403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f88e8006fd0 tx=0x7f88e8006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:26.741 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:26.076+0000 7f8904400700 1 -- 172.21.15.12:0/1611760593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 msgr2=0x7f88fc19ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:26.741 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:26.076+0000 7f8904400700 1 --2- 172.21.15.12:0/1611760593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 0x7f88fc19ddb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f88f0004350 tx=0x7f88f000bd50 comp rx=0 tx=0).stop 2024-03-17T23:46:26.741 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:26.076+0000 7f8904400700 1 -- 172.21.15.12:0/1611760593 shutdown_connections 2024-03-17T23:46:26.741 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:26.076+0000 7f8904400700 1 --2- 172.21.15.12:0/1611760593 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f88e4038f20 0x7f88e403b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:26.741 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:26.076+0000 7f8904400700 1 --2- 172.21.15.12:0/1611760593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88fc106530 0x7f88fc19ddb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:26.741 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:26.076+0000 7f8904400700 1 -- 172.21.15.12:0/1611760593 >> 172.21.15.12:0/1611760593 conn(0x7f88fc102130 msgr2=0x7f88fc1949f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:26.741 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:26.076+0000 7f8904400700 1 -- 172.21.15.12:0/1611760593 shutdown_connections 2024-03-17T23:46:26.741 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:26.076+0000 7f8904400700 1 -- 172.21.15.12:0/1611760593 wait complete. 2024-03-17T23:46:27.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:27 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' 2024-03-17T23:46:27.345 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:27 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' 2024-03-17T23:46:28.301 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:28 smithi012 ceph-mon[31541]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:46:28.317 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOxBkCPX3tVGgoNag9XTh7NfPmwRKi8uPR3kx95S5gsS8mq26AcRbk2GjbnofOpger2V8+1QBrrf3ATtwO9K6VdFHH62B4at28pDel5r1smBZIasWp6f2KR9z/j1QPJbFGlQPBgFZ4nlvLxWUe9KjKxMngCIGTOJa23BQ5JR0+sJchid2r003pfpSmDsnKKRUs7RdDskCuF84owTDqqUSGFRsPzDMytmlJwqO50AstRf/lfMy6filOFo4E224dJ6ld4Qw5LhfCpumeAVvrtVCezp/CkV9oAiH6T66vwO6rtdr2UjPW2NtNJL28IAQw1WTbiwx1jKwyKpdsyGvttEX2j6Anj6g7A6Tr0MULPui7gHBynv5NcZsY9xvfS+DE6+e2eNpPAWDpCjUQuKZikxnv/1kF1bOaSThnSRXjQP9UBgn2nMZJbTO9C0RA/+tnGmU/KGVcxTY7/kiTbrd4wtyME/9317QbZtCns9VmeSNtbVCcKrOh2nbTU10YrmWywis= ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:46:28.318 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.343+0000 7f6203000700 1 Processor -- start 2024-03-17T23:46:28.318 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.344+0000 7f6203000700 1 -- start start 2024-03-17T23:46:28.318 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.344+0000 7f6203000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 0x7f61fc108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:28.319 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.344+0000 7f6203000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc108640 con 0x7f61fc107c50 2024-03-17T23:46:28.319 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.345+0000 7f61fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 0x7f61fc108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:28.319 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.345+0000 7f61fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 0x7f61fc108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50440/0 (socket says 172.21.15.12:50440) 2024-03-17T23:46:28.319 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.345+0000 7f61fbe00700 1 -- 172.21.15.12:0/2190123054 learned_addr learned my addr 172.21.15.12:0/2190123054 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:28.319 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.345+0000 7f61fbe00700 1 -- 172.21.15.12:0/2190123054 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc108e50 con 0x7f61fc107c50 2024-03-17T23:46:28.319 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.345+0000 7f61fbe00700 1 --2- 172.21.15.12:0/2190123054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 0x7f61fc108070 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0098a0 tx=0x7f61ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1830c25ff18faa37 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:28.319 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.346+0000 7f61faa00700 1 -- 172.21.15.12:0/2190123054 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61ec004160 con 0x7f61fc107c50 2024-03-17T23:46:28.319 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.346+0000 7f61faa00700 1 -- 172.21.15.12:0/2190123054 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f61ec004300 con 0x7f61fc107c50 2024-03-17T23:46:28.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.346+0000 7f61faa00700 1 -- 172.21.15.12:0/2190123054 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61ec011b50 con 0x7f61fc107c50 2024-03-17T23:46:28.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.347+0000 7f6203000700 1 -- 172.21.15.12:0/2190123054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 msgr2=0x7f61fc108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:28.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.347+0000 7f6203000700 1 --2- 172.21.15.12:0/2190123054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 0x7f61fc108070 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0098a0 tx=0x7f61ec009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:28.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.347+0000 7f6203000700 1 -- 172.21.15.12:0/2190123054 shutdown_connections 2024-03-17T23:46:28.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.347+0000 7f6203000700 1 --2- 172.21.15.12:0/2190123054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 0x7f61fc108070 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:28.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.347+0000 7f6203000700 1 -- 172.21.15.12:0/2190123054 >> 172.21.15.12:0/2190123054 conn(0x7f61fc103210 msgr2=0x7f61fc105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:28.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.347+0000 7f6203000700 1 -- 172.21.15.12:0/2190123054 shutdown_connections 2024-03-17T23:46:28.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.347+0000 7f6203000700 1 -- 172.21.15.12:0/2190123054 wait complete. 2024-03-17T23:46:28.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.348+0000 7f6203000700 1 Processor -- start 2024-03-17T23:46:28.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.348+0000 7f6203000700 1 -- start start 2024-03-17T23:46:28.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.349+0000 7f6203000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 0x7f61fc07db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:28.322 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.349+0000 7f6203000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc10c200 con 0x7f61fc107c50 2024-03-17T23:46:28.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.349+0000 7f61fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 0x7f61fc07db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:28.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.349+0000 7f61fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 0x7f61fc07db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50442/0 (socket says 172.21.15.12:50442) 2024-03-17T23:46:28.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.349+0000 7f61fbe00700 1 -- 172.21.15.12:0/3286423769 learned_addr learned my addr 172.21.15.12:0/3286423769 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:28.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.349+0000 7f61fbe00700 1 -- 172.21.15.12:0/3286423769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc07e0b0 con 0x7f61fc107c50 2024-03-17T23:46:28.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.350+0000 7f61fbe00700 1 --2- 172.21.15.12:0/3286423769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 0x7f61fc07db70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f61ec000c00 tx=0x7f61ec011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:28.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.350+0000 7f61f8c00700 1 -- 172.21.15.12:0/3286423769 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61ec003b20 con 0x7f61fc107c50 2024-03-17T23:46:28.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.350+0000 7f61f8c00700 1 -- 172.21.15.12:0/3286423769 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f61ec003cc0 con 0x7f61fc107c50 2024-03-17T23:46:28.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.350+0000 7f61f8c00700 1 -- 172.21.15.12:0/3286423769 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61ec024910 con 0x7f61fc107c50 2024-03-17T23:46:28.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.350+0000 7f6203000700 1 -- 172.21.15.12:0/3286423769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61fc07c2a0 con 0x7f61fc107c50 2024-03-17T23:46:28.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.350+0000 7f6203000700 1 -- 172.21.15.12:0/3286423769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61fc07c7e0 con 0x7f61fc107c50 2024-03-17T23:46:28.323 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.351+0000 7f6203000700 1 -- 172.21.15.12:0/3286423769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61fc063900 con 0x7f61fc107c50 2024-03-17T23:46:28.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.351+0000 7f61f8c00700 1 -- 172.21.15.12:0/3286423769 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f61ec022020 con 0x7f61fc107c50 2024-03-17T23:46:28.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.352+0000 7f61f8c00700 1 --2- 172.21.15.12:0/3286423769 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f61e0038fd0 0x7f61e003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:28.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.352+0000 7f61f8c00700 1 -- 172.21.15.12:0/3286423769 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f61ec011e00 con 0x7f61fc107c50 2024-03-17T23:46:28.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.352+0000 7f61fb400700 1 --2- 172.21.15.12:0/3286423769 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f61e0038fd0 0x7f61e003b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:28.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.353+0000 7f61fb400700 1 --2- 172.21.15.12:0/3286423769 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f61e0038fd0 0x7f61e003b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f61e8006fd0 tx=0x7f61e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:28.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.358+0000 7f61f8c00700 1 -- 172.21.15.12:0/3286423769 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f61ec016260 con 0x7f61fc107c50 2024-03-17T23:46:28.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.584+0000 7f6203000700 1 -- 172.21.15.12:0/3286423769 --> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f61fc1956d0 con 0x7f61e0038fd0 2024-03-17T23:46:28.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.586+0000 7f61f8c00700 1 -- 172.21.15.12:0/3286423769 <== mgr.14120 v2:172.21.15.12:6800/2526360059 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f61fc1956d0 con 0x7f61e0038fd0 2024-03-17T23:46:28.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.588+0000 7f6203000700 1 -- 172.21.15.12:0/3286423769 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f61e0038fd0 msgr2=0x7f61e003b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:28.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.588+0000 7f6203000700 1 --2- 172.21.15.12:0/3286423769 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f61e0038fd0 0x7f61e003b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f61e8006fd0 tx=0x7f61e8006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:28.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.588+0000 7f6203000700 1 -- 172.21.15.12:0/3286423769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 msgr2=0x7f61fc07db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:28.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.588+0000 7f6203000700 1 --2- 172.21.15.12:0/3286423769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 0x7f61fc07db70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f61ec000c00 tx=0x7f61ec011c10 comp rx=0 tx=0).stop 2024-03-17T23:46:28.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.589+0000 7f6203000700 1 -- 172.21.15.12:0/3286423769 shutdown_connections 2024-03-17T23:46:28.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.589+0000 7f6203000700 1 --2- 172.21.15.12:0/3286423769 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f61e0038fd0 0x7f61e003b490 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:28.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.589+0000 7f6203000700 1 --2- 172.21.15.12:0/3286423769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc107c50 0x7f61fc07db70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:28.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.589+0000 7f6203000700 1 -- 172.21.15.12:0/3286423769 >> 172.21.15.12:0/3286423769 conn(0x7f61fc103210 msgr2=0x7f61fc103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:28.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.589+0000 7f6203000700 1 -- 172.21.15.12:0/3286423769 shutdown_connections 2024-03-17T23:46:28.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:27.589+0000 7f6203000700 1 -- 172.21.15.12:0/3286423769 wait complete. 2024-03-17T23:46:28.327 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-17T23:46:28.327 INFO:teuthology.orchestra.run.smithi012.stdout:Adding key to root@localhost authorized_keys... 2024-03-17T23:46:28.327 INFO:teuthology.orchestra.run.smithi012.stdout:Adding host smithi012... 2024-03-17T23:46:29.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:29 smithi012 ceph-mon[31541]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi012", "addr": "172.21.15.12", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:46:30.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:30 smithi012 ceph-mon[31541]: Deploying cephadm binary to smithi012 2024-03-17T23:46:36.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Added host 'smithi012' with addr '172.21.15.12' 2024-03-17T23:46:36.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.943+0000 7f855c000700 1 Processor -- start 2024-03-17T23:46:36.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.944+0000 7f855c000700 1 -- start start 2024-03-17T23:46:36.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.944+0000 7f855c000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 0x7f8554108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:36.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.944+0000 7f855c000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8554108640 con 0x7f8554107c50 2024-03-17T23:46:36.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.945+0000 7f8559600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 0x7f8554108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:36.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.945+0000 7f8559600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 0x7f8554108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50456/0 (socket says 172.21.15.12:50456) 2024-03-17T23:46:36.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.945+0000 7f8559600700 1 -- 172.21.15.12:0/1110513929 learned_addr learned my addr 172.21.15.12:0/1110513929 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:36.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.945+0000 7f8559600700 1 -- 172.21.15.12:0/1110513929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8554108e50 con 0x7f8554107c50 2024-03-17T23:46:36.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.946+0000 7f8559600700 1 --2- 172.21.15.12:0/1110513929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 0x7f8554108070 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f85480098a0 tx=0x7f8548009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913aaddc81f077c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:36.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.946+0000 7f8553e00700 1 -- 172.21.15.12:0/1110513929 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8548004160 con 0x7f8554107c50 2024-03-17T23:46:36.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.946+0000 7f8553e00700 1 -- 172.21.15.12:0/1110513929 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8548004300 con 0x7f8554107c50 2024-03-17T23:46:36.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.946+0000 7f8553e00700 1 -- 172.21.15.12:0/1110513929 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8548011b50 con 0x7f8554107c50 2024-03-17T23:46:36.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.947+0000 7f855c000700 1 -- 172.21.15.12:0/1110513929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 msgr2=0x7f8554108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:36.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.947+0000 7f855c000700 1 --2- 172.21.15.12:0/1110513929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 0x7f8554108070 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f85480098a0 tx=0x7f8548009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:36.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.947+0000 7f855c000700 1 -- 172.21.15.12:0/1110513929 shutdown_connections 2024-03-17T23:46:36.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.947+0000 7f855c000700 1 --2- 172.21.15.12:0/1110513929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 0x7f8554108070 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:36.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.947+0000 7f855c000700 1 -- 172.21.15.12:0/1110513929 >> 172.21.15.12:0/1110513929 conn(0x7f85541031f0 msgr2=0x7f8554105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:36.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.947+0000 7f855c000700 1 -- 172.21.15.12:0/1110513929 shutdown_connections 2024-03-17T23:46:36.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.947+0000 7f855c000700 1 -- 172.21.15.12:0/1110513929 wait complete. 2024-03-17T23:46:36.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.948+0000 7f855c000700 1 Processor -- start 2024-03-17T23:46:36.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.948+0000 7f855c000700 1 -- start start 2024-03-17T23:46:36.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.948+0000 7f855c000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 0x7f855407ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:36.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.948+0000 7f855c000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f855410c200 con 0x7f8554107c50 2024-03-17T23:46:36.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.949+0000 7f8559600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 0x7f855407ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:36.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.949+0000 7f8559600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 0x7f855407ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50458/0 (socket says 172.21.15.12:50458) 2024-03-17T23:46:36.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.949+0000 7f8559600700 1 -- 172.21.15.12:0/3319242521 learned_addr learned my addr 172.21.15.12:0/3319242521 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:36.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.949+0000 7f8559600700 1 -- 172.21.15.12:0/3319242521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855407cfa0 con 0x7f8554107c50 2024-03-17T23:46:36.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.949+0000 7f8559600700 1 --2- 172.21.15.12:0/3319242521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 0x7f855407ca60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8548000c00 tx=0x7f8548011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:36.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.950+0000 7f8552000700 1 -- 172.21.15.12:0/3319242521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8548003b20 con 0x7f8554107c50 2024-03-17T23:46:36.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.950+0000 7f8552000700 1 -- 172.21.15.12:0/3319242521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8548003cc0 con 0x7f8554107c50 2024-03-17T23:46:36.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.950+0000 7f855c000700 1 -- 172.21.15.12:0/3319242521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f855407b190 con 0x7f8554107c50 2024-03-17T23:46:36.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.950+0000 7f8552000700 1 -- 172.21.15.12:0/3319242521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8548024910 con 0x7f8554107c50 2024-03-17T23:46:36.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.950+0000 7f855c000700 1 -- 172.21.15.12:0/3319242521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f855407b6d0 con 0x7f8554107c50 2024-03-17T23:46:36.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.951+0000 7f855c000700 1 -- 172.21.15.12:0/3319242521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8554063900 con 0x7f8554107c50 2024-03-17T23:46:36.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.951+0000 7f8552000700 1 -- 172.21.15.12:0/3319242521 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f8548022020 con 0x7f8554107c50 2024-03-17T23:46:36.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.952+0000 7f8552000700 1 --2- 172.21.15.12:0/3319242521 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f853c038f20 0x7f853c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:36.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.952+0000 7f8552000700 1 -- 172.21.15.12:0/3319242521 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8548011e00 con 0x7f8554107c50 2024-03-17T23:46:36.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.952+0000 7f8558c00700 1 --2- 172.21.15.12:0/3319242521 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f853c038f20 0x7f853c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:36.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.953+0000 7f8558c00700 1 --2- 172.21.15.12:0/3319242521 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f853c038f20 0x7f853c03b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8540006fd0 tx=0x7f8540006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:36.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:28.958+0000 7f8552000700 1 -- 172.21.15.12:0/3319242521 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8548016260 con 0x7f8554107c50 2024-03-17T23:46:36.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:29.185+0000 7f855c000700 1 -- 172.21.15.12:0/3319242521 --> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi012", "addr": "172.21.15.12", "target": ["mon-mgr", ""]}) -- 0x7f8554104500 con 0x7f853c038f20 2024-03-17T23:46:36.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:35.567+0000 7f8552000700 1 -- 172.21.15.12:0/3319242521 <== mgr.14120 v2:172.21.15.12:6800/2526360059 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f8554104500 con 0x7f853c038f20 2024-03-17T23:46:36.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:35.572+0000 7f855c000700 1 -- 172.21.15.12:0/3319242521 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f853c038f20 msgr2=0x7f853c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:36.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:35.572+0000 7f855c000700 1 --2- 172.21.15.12:0/3319242521 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f853c038f20 0x7f853c03b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8540006fd0 tx=0x7f8540006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:36.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:35.572+0000 7f855c000700 1 -- 172.21.15.12:0/3319242521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 msgr2=0x7f855407ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:36.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:35.572+0000 7f855c000700 1 --2- 172.21.15.12:0/3319242521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 0x7f855407ca60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8548000c00 tx=0x7f8548011c10 comp rx=0 tx=0).stop 2024-03-17T23:46:36.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:35.572+0000 7f855c000700 1 -- 172.21.15.12:0/3319242521 shutdown_connections 2024-03-17T23:46:36.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:35.572+0000 7f855c000700 1 --2- 172.21.15.12:0/3319242521 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f853c038f20 0x7f853c03b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:36.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:35.572+0000 7f855c000700 1 --2- 172.21.15.12:0/3319242521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8554107c50 0x7f855407ca60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:36.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:35.572+0000 7f855c000700 1 -- 172.21.15.12:0/3319242521 >> 172.21.15.12:0/3319242521 conn(0x7f85541031f0 msgr2=0x7f8554103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:36.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:35.573+0000 7f855c000700 1 -- 172.21.15.12:0/3319242521 shutdown_connections 2024-03-17T23:46:36.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:35.573+0000 7f855c000700 1 -- 172.21.15.12:0/3319242521 wait complete. 2024-03-17T23:46:36.305 INFO:teuthology.orchestra.run.smithi012.stdout:Deploying unmanaged mon service... 2024-03-17T23:46:36.820 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:36 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' 2024-03-17T23:46:36.821 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:36 smithi012 ceph-mon[31541]: Added host smithi012 2024-03-17T23:46:36.821 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:36 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:46:38.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-17T23:46:38.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.982+0000 7febefe00700 1 Processor -- start 2024-03-17T23:46:38.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.982+0000 7febefe00700 1 -- start start 2024-03-17T23:46:38.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.983+0000 7febefe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 0x7febe81089a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:38.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.983+0000 7febefe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7febe807beb0 con 0x7febe8106570 2024-03-17T23:46:38.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.983+0000 7febed400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 0x7febe81089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:38.022 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.983+0000 7febed400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 0x7febe81089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47428/0 (socket says 172.21.15.12:47428) 2024-03-17T23:46:38.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.983+0000 7febed400700 1 -- 172.21.15.12:0/3333789466 learned_addr learned my addr 172.21.15.12:0/3333789466 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:38.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.984+0000 7febed400700 1 -- 172.21.15.12:0/3333789466 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe8108ee0 con 0x7febe8106570 2024-03-17T23:46:38.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.984+0000 7febed400700 1 --2- 172.21.15.12:0/3333789466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 0x7febe81089a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7febdc0098a0 tx=0x7febdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f9534f57b4823f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:38.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.984+0000 7febe7e00700 1 -- 172.21.15.12:0/3333789466 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febdc004160 con 0x7febe8106570 2024-03-17T23:46:38.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.984+0000 7febe7e00700 1 -- 172.21.15.12:0/3333789466 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7febdc004300 con 0x7febe8106570 2024-03-17T23:46:38.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.984+0000 7febe7e00700 1 -- 172.21.15.12:0/3333789466 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febdc011b50 con 0x7febe8106570 2024-03-17T23:46:38.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.985+0000 7febefe00700 1 -- 172.21.15.12:0/3333789466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 msgr2=0x7febe81089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:38.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.985+0000 7febefe00700 1 --2- 172.21.15.12:0/3333789466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 0x7febe81089a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7febdc0098a0 tx=0x7febdc009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:38.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.985+0000 7febefe00700 1 -- 172.21.15.12:0/3333789466 shutdown_connections 2024-03-17T23:46:38.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.985+0000 7febefe00700 1 --2- 172.21.15.12:0/3333789466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 0x7febe81089a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:38.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.985+0000 7febefe00700 1 -- 172.21.15.12:0/3333789466 >> 172.21.15.12:0/3333789466 conn(0x7febe8102190 msgr2=0x7febe81045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:38.023 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.986+0000 7febefe00700 1 -- 172.21.15.12:0/3333789466 shutdown_connections 2024-03-17T23:46:38.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.986+0000 7febefe00700 1 -- 172.21.15.12:0/3333789466 wait complete. 2024-03-17T23:46:38.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.987+0000 7febefe00700 1 Processor -- start 2024-03-17T23:46:38.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.987+0000 7febefe00700 1 -- start start 2024-03-17T23:46:38.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.987+0000 7febefe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 0x7febe81a2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:38.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.987+0000 7febefe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7febe8109f70 con 0x7febe8106570 2024-03-17T23:46:38.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.987+0000 7febed400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 0x7febe81a2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:38.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.987+0000 7febed400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 0x7febe81a2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47432/0 (socket says 172.21.15.12:47432) 2024-03-17T23:46:38.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.987+0000 7febed400700 1 -- 172.21.15.12:0/2691350929 learned_addr learned my addr 172.21.15.12:0/2691350929 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:38.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.988+0000 7febed400700 1 -- 172.21.15.12:0/2691350929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe81a2ad0 con 0x7febe8106570 2024-03-17T23:46:38.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.988+0000 7febed400700 1 --2- 172.21.15.12:0/2691350929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 0x7febe81a2590 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febdc011b60 tx=0x7febdc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:38.027 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.988+0000 7febe6000700 1 -- 172.21.15.12:0/2691350929 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febdc003b20 con 0x7febe8106570 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.988+0000 7febe6000700 1 -- 172.21.15.12:0/2691350929 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7febdc003cc0 con 0x7febe8106570 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.988+0000 7febe6000700 1 -- 172.21.15.12:0/2691350929 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febdc024840 con 0x7febe8106570 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.988+0000 7febefe00700 1 -- 172.21.15.12:0/2691350929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe81a2d60 con 0x7febe8106570 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.989+0000 7febefe00700 1 -- 172.21.15.12:0/2691350929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe81a5a50 con 0x7febe8106570 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.990+0000 7febe6000700 1 -- 172.21.15.12:0/2691350929 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7febdc022020 con 0x7febe8106570 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.990+0000 7febefe00700 1 -- 172.21.15.12:0/2691350929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe8063900 con 0x7febe8106570 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.991+0000 7febe6000700 1 --2- 172.21.15.12:0/2691350929 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7febd0038fd0 0x7febd003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.991+0000 7febe6000700 1 -- 172.21.15.12:0/2691350929 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7febdc011e00 con 0x7febe8106570 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.991+0000 7febeca00700 1 --2- 172.21.15.12:0/2691350929 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7febd0038fd0 0x7febd003b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.992+0000 7febeca00700 1 --2- 172.21.15.12:0/2691350929 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7febd0038fd0 0x7febd003b490 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7febd4006fd0 tx=0x7febd4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:36.997+0000 7febe6000700 1 -- 172.21.15.12:0/2691350929 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7febdc00b3e0 con 0x7febe8106570 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:37.272+0000 7febefe00700 1 -- 172.21.15.12:0/2691350929 --> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7febe810ae50 con 0x7febd0038fd0 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:37.280+0000 7febe6000700 1 -- 172.21.15.12:0/2691350929 <== mgr.14120 v2:172.21.15.12:6800/2526360059 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7febe810ae50 con 0x7febd0038fd0 2024-03-17T23:46:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:37.283+0000 7febefe00700 1 -- 172.21.15.12:0/2691350929 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7febd0038fd0 msgr2=0x7febd003b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:38.029 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:37.283+0000 7febefe00700 1 --2- 172.21.15.12:0/2691350929 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7febd0038fd0 0x7febd003b490 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7febd4006fd0 tx=0x7febd4006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:38.029 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:37.283+0000 7febefe00700 1 -- 172.21.15.12:0/2691350929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 msgr2=0x7febe81a2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:38.029 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:37.283+0000 7febefe00700 1 --2- 172.21.15.12:0/2691350929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 0x7febe81a2590 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febdc011b60 tx=0x7febdc011c40 comp rx=0 tx=0).stop 2024-03-17T23:46:38.029 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:37.283+0000 7febefe00700 1 -- 172.21.15.12:0/2691350929 shutdown_connections 2024-03-17T23:46:38.029 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:37.283+0000 7febefe00700 1 --2- 172.21.15.12:0/2691350929 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7febd0038fd0 0x7febd003b490 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:38.030 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:37.283+0000 7febefe00700 1 --2- 172.21.15.12:0/2691350929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febe8106570 0x7febe81a2590 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:38.030 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:37.283+0000 7febefe00700 1 -- 172.21.15.12:0/2691350929 >> 172.21.15.12:0/2691350929 conn(0x7febe8102190 msgr2=0x7febe81045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:38.030 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:37.283+0000 7febefe00700 1 -- 172.21.15.12:0/2691350929 shutdown_connections 2024-03-17T23:46:38.030 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:37.283+0000 7febefe00700 1 -- 172.21.15.12:0/2691350929 wait complete. 2024-03-17T23:46:38.030 INFO:teuthology.orchestra.run.smithi012.stdout:Deploying unmanaged mgr service... 2024-03-17T23:46:38.542 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:38 smithi012 ceph-mon[31541]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:46:38.542 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:38 smithi012 ceph-mon[31541]: Saving service mon spec with placement count:5 2024-03-17T23:46:38.543 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:38 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' 2024-03-17T23:46:40.220 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-17T23:46:40.220 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.854+0000 7f91ccc00700 1 Processor -- start 2024-03-17T23:46:40.220 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.854+0000 7f91ccc00700 1 -- start start 2024-03-17T23:46:40.220 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.854+0000 7f91ccc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 0x7f91c8108340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:40.221 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.854+0000 7f91ccc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f91c8108910 con 0x7f91c8107f40 2024-03-17T23:46:40.221 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.854+0000 7f91c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 0x7f91c8108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:40.221 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.855+0000 7f91c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 0x7f91c8108340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47440/0 (socket says 172.21.15.12:47440) 2024-03-17T23:46:40.221 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.855+0000 7f91c7400700 1 -- 172.21.15.12:0/3260286930 learned_addr learned my addr 172.21.15.12:0/3260286930 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:40.221 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.855+0000 7f91c7400700 1 -- 172.21.15.12:0/3260286930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c8108a90 con 0x7f91c8107f40 2024-03-17T23:46:40.221 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.855+0000 7f91c7400700 1 --2- 172.21.15.12:0/3260286930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 0x7f91c8108340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f91b80098a0 tx=0x7f91b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4680d8c6502d4a26 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:40.221 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.856+0000 7f91c6000700 1 -- 172.21.15.12:0/3260286930 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91b8004160 con 0x7f91c8107f40 2024-03-17T23:46:40.221 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.856+0000 7f91c6000700 1 -- 172.21.15.12:0/3260286930 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f91b8004300 con 0x7f91c8107f40 2024-03-17T23:46:40.221 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.856+0000 7f91c6000700 1 -- 172.21.15.12:0/3260286930 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91b8011b50 con 0x7f91c8107f40 2024-03-17T23:46:40.221 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.856+0000 7f91ccc00700 1 -- 172.21.15.12:0/3260286930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 msgr2=0x7f91c8108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:40.221 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.856+0000 7f91ccc00700 1 --2- 172.21.15.12:0/3260286930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 0x7f91c8108340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f91b80098a0 tx=0x7f91b8009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:40.221 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.856+0000 7f91ccc00700 1 -- 172.21.15.12:0/3260286930 shutdown_connections 2024-03-17T23:46:40.221 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.856+0000 7f91ccc00700 1 --2- 172.21.15.12:0/3260286930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 0x7f91c8108340 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.856+0000 7f91ccc00700 1 -- 172.21.15.12:0/3260286930 >> 172.21.15.12:0/3260286930 conn(0x7f91c8101690 msgr2=0x7f91c8103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.856+0000 7f91ccc00700 1 -- 172.21.15.12:0/3260286930 shutdown_connections 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.856+0000 7f91ccc00700 1 -- 172.21.15.12:0/3260286930 wait complete. 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.857+0000 7f91ccc00700 1 Processor -- start 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.858+0000 7f91ccc00700 1 -- start start 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.858+0000 7f91ccc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 0x7f91c819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.858+0000 7f91ccc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f91c8106970 con 0x7f91c8107f40 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.858+0000 7f91c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 0x7f91c819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.858+0000 7f91c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 0x7f91c819dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47442/0 (socket says 172.21.15.12:47442) 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.858+0000 7f91c7400700 1 -- 172.21.15.12:0/3587369100 learned_addr learned my addr 172.21.15.12:0/3587369100 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.859+0000 7f91c7400700 1 -- 172.21.15.12:0/3587369100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c819e4f0 con 0x7f91c8107f40 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.859+0000 7f91c7400700 1 --2- 172.21.15.12:0/3587369100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 0x7f91c819dfb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f91b8011670 tx=0x7f91b8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.859+0000 7f91bfe00700 1 -- 172.21.15.12:0/3587369100 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91b80039c0 con 0x7f91c8107f40 2024-03-17T23:46:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.859+0000 7f91bfe00700 1 -- 172.21.15.12:0/3587369100 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f91b8011e50 con 0x7f91c8107f40 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.859+0000 7f91ccc00700 1 -- 172.21.15.12:0/3587369100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91c819e780 con 0x7f91c8107f40 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.859+0000 7f91bfe00700 1 -- 172.21.15.12:0/3587369100 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91b801a5c0 con 0x7f91c8107f40 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.859+0000 7f91ccc00700 1 -- 172.21.15.12:0/3587369100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91c81a1470 con 0x7f91c8107f40 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.860+0000 7f91ccc00700 1 -- 172.21.15.12:0/3587369100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91c8063900 con 0x7f91c8107f40 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.860+0000 7f91bfe00700 1 -- 172.21.15.12:0/3587369100 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f91b8024430 con 0x7f91c8107f40 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.861+0000 7f91bfe00700 1 --2- 172.21.15.12:0/3587369100 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f91ac038f20 0x7f91ac03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.861+0000 7f91bfe00700 1 -- 172.21.15.12:0/3587369100 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f91b804d210 con 0x7f91c8107f40 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.861+0000 7f91c6a00700 1 --2- 172.21.15.12:0/3587369100 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f91ac038f20 0x7f91ac03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.862+0000 7f91c6a00700 1 --2- 172.21.15.12:0/3587369100 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f91ac038f20 0x7f91ac03b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f91b4006fd0 tx=0x7f91b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:38.867+0000 7f91bfe00700 1 -- 172.21.15.12:0/3587369100 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f91b8017ca0 con 0x7f91c8107f40 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:39.108+0000 7f91ccc00700 1 -- 172.21.15.12:0/3587369100 --> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f91c8062710 con 0x7f91ac038f20 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:39.118+0000 7f91bfe00700 1 -- 172.21.15.12:0/3587369100 <== mgr.14120 v2:172.21.15.12:6800/2526360059 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f91c8062710 con 0x7f91ac038f20 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:39.120+0000 7f91ccc00700 1 -- 172.21.15.12:0/3587369100 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f91ac038f20 msgr2=0x7f91ac03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:40.223 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:39.120+0000 7f91ccc00700 1 --2- 172.21.15.12:0/3587369100 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f91ac038f20 0x7f91ac03b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f91b4006fd0 tx=0x7f91b4006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:40.224 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:39.120+0000 7f91ccc00700 1 -- 172.21.15.12:0/3587369100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 msgr2=0x7f91c819dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:40.224 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:39.120+0000 7f91ccc00700 1 --2- 172.21.15.12:0/3587369100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 0x7f91c819dfb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f91b8011670 tx=0x7f91b8011b60 comp rx=0 tx=0).stop 2024-03-17T23:46:40.224 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:39.120+0000 7f91ccc00700 1 -- 172.21.15.12:0/3587369100 shutdown_connections 2024-03-17T23:46:40.224 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:39.121+0000 7f91ccc00700 1 --2- 172.21.15.12:0/3587369100 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f91ac038f20 0x7f91ac03b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:40.224 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:39.121+0000 7f91ccc00700 1 --2- 172.21.15.12:0/3587369100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91c8107f40 0x7f91c819dfb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:40.224 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:39.121+0000 7f91ccc00700 1 -- 172.21.15.12:0/3587369100 >> 172.21.15.12:0/3587369100 conn(0x7f91c8101690 msgr2=0x7f91c81945d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:40.224 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:39.121+0000 7f91ccc00700 1 -- 172.21.15.12:0/3587369100 shutdown_connections 2024-03-17T23:46:40.224 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:39.121+0000 7f91ccc00700 1 -- 172.21.15.12:0/3587369100 wait complete. 2024-03-17T23:46:40.391 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:40 smithi012 ceph-mon[31541]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:46:40.391 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:40 smithi012 ceph-mon[31541]: Saving service mgr spec with placement count:2 2024-03-17T23:46:40.391 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:40 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' 2024-03-17T23:46:40.391 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:40 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' 2024-03-17T23:46:41.676 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.779+0000 7f0eb2a00700 1 Processor -- start 2024-03-17T23:46:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.780+0000 7f0eb2a00700 1 -- start start 2024-03-17T23:46:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.780+0000 7f0eb2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 0x7f0eac108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.780+0000 7f0eb2a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac108640 con 0x7f0eac107c50 2024-03-17T23:46:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.780+0000 7f0eabe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 0x7f0eac108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.780+0000 7f0eabe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 0x7f0eac108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47450/0 (socket says 172.21.15.12:47450) 2024-03-17T23:46:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.780+0000 7f0eabe00700 1 -- 172.21.15.12:0/2295086854 learned_addr learned my addr 172.21.15.12:0/2295086854 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.781+0000 7f0eabe00700 1 -- 172.21.15.12:0/2295086854 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac108e50 con 0x7f0eac107c50 2024-03-17T23:46:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.781+0000 7f0eabe00700 1 --2- 172.21.15.12:0/2295086854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 0x7f0eac108070 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0098a0 tx=0x7f0e9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d12621faf88d7da server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.781+0000 7f0eaaa00700 1 -- 172.21.15.12:0/2295086854 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e9c004160 con 0x7f0eac107c50 2024-03-17T23:46:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.781+0000 7f0eaaa00700 1 -- 172.21.15.12:0/2295086854 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e9c004300 con 0x7f0eac107c50 2024-03-17T23:46:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.781+0000 7f0eaaa00700 1 -- 172.21.15.12:0/2295086854 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e9c011b20 con 0x7f0eac107c50 2024-03-17T23:46:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.782+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2295086854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 msgr2=0x7f0eac108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.782+0000 7f0eb2a00700 1 --2- 172.21.15.12:0/2295086854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 0x7f0eac108070 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0098a0 tx=0x7f0e9c009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.782+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2295086854 shutdown_connections 2024-03-17T23:46:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.782+0000 7f0eb2a00700 1 --2- 172.21.15.12:0/2295086854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 0x7f0eac108070 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.782+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2295086854 >> 172.21.15.12:0/2295086854 conn(0x7f0eac1031f0 msgr2=0x7f0eac105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.782+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2295086854 shutdown_connections 2024-03-17T23:46:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.783+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2295086854 wait complete. 2024-03-17T23:46:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.784+0000 7f0eb2a00700 1 Processor -- start 2024-03-17T23:46:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.784+0000 7f0eb2a00700 1 -- start start 2024-03-17T23:46:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.784+0000 7f0eb2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 0x7f0eac07ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.784+0000 7f0eb2a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac10c200 con 0x7f0eac107c50 2024-03-17T23:46:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.784+0000 7f0eabe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 0x7f0eac07ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.784+0000 7f0eabe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 0x7f0eac07ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47454/0 (socket says 172.21.15.12:47454) 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.784+0000 7f0eabe00700 1 -- 172.21.15.12:0/2729547378 learned_addr learned my addr 172.21.15.12:0/2729547378 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.785+0000 7f0eabe00700 1 -- 172.21.15.12:0/2729547378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac07cfa0 con 0x7f0eac107c50 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.785+0000 7f0eabe00700 1 --2- 172.21.15.12:0/2729547378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 0x7f0eac07ca60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0117e0 tx=0x7f0e9c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.785+0000 7f0ea8c00700 1 -- 172.21.15.12:0/2729547378 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e9c003bf0 con 0x7f0eac107c50 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.786+0000 7f0ea8c00700 1 -- 172.21.15.12:0/2729547378 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e9c01a910 con 0x7f0eac107c50 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.786+0000 7f0ea8c00700 1 -- 172.21.15.12:0/2729547378 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e9c024970 con 0x7f0eac107c50 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.786+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2729547378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eac07b190 con 0x7f0eac107c50 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.786+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2729547378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eac07b650 con 0x7f0eac107c50 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.787+0000 7f0ea8c00700 1 -- 172.21.15.12:0/2729547378 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f0e9c022020 con 0x7f0eac107c50 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.787+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2729547378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eac063900 con 0x7f0eac107c50 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.787+0000 7f0ea8c00700 1 --2- 172.21.15.12:0/2729547378 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f0e90038f70 0x7f0e9003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.788+0000 7f0ea8c00700 1 -- 172.21.15.12:0/2729547378 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0e9c04d250 con 0x7f0eac107c50 2024-03-17T23:46:41.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.788+0000 7f0eab400700 1 --2- 172.21.15.12:0/2729547378 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f0e90038f70 0x7f0e9003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.789+0000 7f0eab400700 1 --2- 172.21.15.12:0/2729547378 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f0e90038f70 0x7f0e9003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0e98006fd0 tx=0x7f0e98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:40.793+0000 7f0ea8c00700 1 -- 172.21.15.12:0/2729547378 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0e9c04cba0 con 0x7f0eac107c50 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:41.023+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2729547378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f0eac07bdc0 con 0x7f0eac107c50 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:41.029+0000 7f0ea8c00700 1 -- 172.21.15.12:0/2729547378 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f0e9c01ab70 con 0x7f0eac107c50 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:41.032+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2729547378 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f0e90038f70 msgr2=0x7f0e9003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:41.032+0000 7f0eb2a00700 1 --2- 172.21.15.12:0/2729547378 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f0e90038f70 0x7f0e9003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0e98006fd0 tx=0x7f0e98006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:41.032+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2729547378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 msgr2=0x7f0eac07ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:41.032+0000 7f0eb2a00700 1 --2- 172.21.15.12:0/2729547378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 0x7f0eac07ca60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0117e0 tx=0x7f0e9c011ba0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:41.032+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2729547378 shutdown_connections 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:41.032+0000 7f0eb2a00700 1 --2- 172.21.15.12:0/2729547378 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f0e90038f70 0x7f0e9003b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:41.032+0000 7f0eb2a00700 1 --2- 172.21.15.12:0/2729547378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eac107c50 0x7f0eac07ca60 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0117e0 tx=0x7f0e9c011ba0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:41.032+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2729547378 >> 172.21.15.12:0/2729547378 conn(0x7f0eac1031f0 msgr2=0x7f0eac103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:41.033+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2729547378 shutdown_connections 2024-03-17T23:46:41.680 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:41.033+0000 7f0eb2a00700 1 -- 172.21.15.12:0/2729547378 wait complete. 2024-03-17T23:46:41.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:41 smithi012 ceph-mon[31541]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:46:41.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:41 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' 2024-03-17T23:46:41.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:41 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2729547378' entity='client.admin' 2024-03-17T23:46:43.613 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.319+0000 7f254d000700 1 Processor -- start 2024-03-17T23:46:43.613 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.319+0000 7f254d000700 1 -- start start 2024-03-17T23:46:43.613 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.320+0000 7f254d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 0x7f254807b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:43.614 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.320+0000 7f254d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f254807b800 con 0x7f254807cdd0 2024-03-17T23:46:43.614 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.320+0000 7f2545e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 0x7f254807b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:43.614 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.320+0000 7f2545e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 0x7f254807b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47458/0 (socket says 172.21.15.12:47458) 2024-03-17T23:46:43.614 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.321+0000 7f2545e00700 1 -- 172.21.15.12:0/4125365536 learned_addr learned my addr 172.21.15.12:0/4125365536 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:43.614 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.321+0000 7f2545e00700 1 -- 172.21.15.12:0/4125365536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254807b980 con 0x7f254807cdd0 2024-03-17T23:46:43.614 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.322+0000 7f2545e00700 1 --2- 172.21.15.12:0/4125365536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 0x7f254807b230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f25380098a0 tx=0x7f2538009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88271119d90d5512 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:43.617 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.322+0000 7f2544a00700 1 -- 172.21.15.12:0/4125365536 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2538004160 con 0x7f254807cdd0 2024-03-17T23:46:43.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.322+0000 7f2544a00700 1 -- 172.21.15.12:0/4125365536 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2538004300 con 0x7f254807cdd0 2024-03-17T23:46:43.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.322+0000 7f2544a00700 1 -- 172.21.15.12:0/4125365536 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2538011b20 con 0x7f254807cdd0 2024-03-17T23:46:43.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.323+0000 7f254d000700 1 -- 172.21.15.12:0/4125365536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 msgr2=0x7f254807b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:43.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.323+0000 7f254d000700 1 --2- 172.21.15.12:0/4125365536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 0x7f254807b230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f25380098a0 tx=0x7f2538009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:43.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.323+0000 7f254d000700 1 -- 172.21.15.12:0/4125365536 shutdown_connections 2024-03-17T23:46:43.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.323+0000 7f254d000700 1 --2- 172.21.15.12:0/4125365536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 0x7f254807b230 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:43.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.323+0000 7f254d000700 1 -- 172.21.15.12:0/4125365536 >> 172.21.15.12:0/4125365536 conn(0x7f2548103290 msgr2=0x7f25481056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:43.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.323+0000 7f254d000700 1 -- 172.21.15.12:0/4125365536 shutdown_connections 2024-03-17T23:46:43.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.323+0000 7f254d000700 1 -- 172.21.15.12:0/4125365536 wait complete. 2024-03-17T23:46:43.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.324+0000 7f254d000700 1 Processor -- start 2024-03-17T23:46:43.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.324+0000 7f254d000700 1 -- start start 2024-03-17T23:46:43.618 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.325+0000 7f254d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 0x7f254819e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.325+0000 7f254d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f254810e320 con 0x7f254807cdd0 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.325+0000 7f2545e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 0x7f254819e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.325+0000 7f2545e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 0x7f254819e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47468/0 (socket says 172.21.15.12:47468) 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.325+0000 7f2545e00700 1 -- 172.21.15.12:0/3629048516 learned_addr learned my addr 172.21.15.12:0/3629048516 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.326+0000 7f2545e00700 1 -- 172.21.15.12:0/3629048516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254819e640 con 0x7f254807cdd0 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.326+0000 7f2545e00700 1 --2- 172.21.15.12:0/3629048516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 0x7f254819e100 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f25380117e0 tx=0x7f2538011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.326+0000 7f253ea00700 1 -- 172.21.15.12:0/3629048516 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2538003bf0 con 0x7f254807cdd0 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.326+0000 7f253ea00700 1 -- 172.21.15.12:0/3629048516 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2538003d90 con 0x7f254807cdd0 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.326+0000 7f253ea00700 1 -- 172.21.15.12:0/3629048516 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2538024800 con 0x7f254807cdd0 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.326+0000 7f254d000700 1 -- 172.21.15.12:0/3629048516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f254819e8d0 con 0x7f254807cdd0 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.327+0000 7f254d000700 1 -- 172.21.15.12:0/3629048516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f254819ed30 con 0x7f254807cdd0 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.328+0000 7f254d000700 1 -- 172.21.15.12:0/3629048516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2548063900 con 0x7f254807cdd0 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.328+0000 7f253ea00700 1 -- 172.21.15.12:0/3629048516 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f2538022020 con 0x7f254807cdd0 2024-03-17T23:46:43.619 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.329+0000 7f253ea00700 1 --2- 172.21.15.12:0/3629048516 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f252c038f70 0x7f252c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:43.620 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.329+0000 7f253ea00700 1 -- 172.21.15.12:0/3629048516 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f253804d130 con 0x7f254807cdd0 2024-03-17T23:46:43.620 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.329+0000 7f2545400700 1 --2- 172.21.15.12:0/3629048516 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f252c038f70 0x7f252c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:43.620 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.330+0000 7f2545400700 1 --2- 172.21.15.12:0/3629048516 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f252c038f70 0x7f252c03b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2534006fd0 tx=0x7f2534006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:43.620 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.334+0000 7f253ea00700 1 -- 172.21.15.12:0/3629048516 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2538016260 con 0x7f254807cdd0 2024-03-17T23:46:43.620 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.585+0000 7f254d000700 1 -- 172.21.15.12:0/3629048516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f254802dce0 con 0x7f254807cdd0 2024-03-17T23:46:43.620 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.590+0000 7f253ea00700 1 -- 172.21.15.12:0/3629048516 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f253801a950 con 0x7f254807cdd0 2024-03-17T23:46:43.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.593+0000 7f254d000700 1 -- 172.21.15.12:0/3629048516 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f252c038f70 msgr2=0x7f252c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:43.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.593+0000 7f254d000700 1 --2- 172.21.15.12:0/3629048516 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f252c038f70 0x7f252c03b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2534006fd0 tx=0x7f2534006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:43.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.593+0000 7f254d000700 1 -- 172.21.15.12:0/3629048516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 msgr2=0x7f254819e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:43.621 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.593+0000 7f254d000700 1 --2- 172.21.15.12:0/3629048516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 0x7f254819e100 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f25380117e0 tx=0x7f2538011ba0 comp rx=0 tx=0).stop 2024-03-17T23:46:43.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.593+0000 7f254d000700 1 -- 172.21.15.12:0/3629048516 shutdown_connections 2024-03-17T23:46:43.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.593+0000 7f254d000700 1 --2- 172.21.15.12:0/3629048516 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f252c038f70 0x7f252c03b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:43.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.593+0000 7f254d000700 1 --2- 172.21.15.12:0/3629048516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254807cdd0 0x7f254819e100 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:43.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.593+0000 7f254d000700 1 -- 172.21.15.12:0/3629048516 >> 172.21.15.12:0/3629048516 conn(0x7f2548103290 msgr2=0x7f2548103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:43.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.594+0000 7f254d000700 1 -- 172.21.15.12:0/3629048516 shutdown_connections 2024-03-17T23:46:43.622 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:42.594+0000 7f254d000700 1 -- 172.21.15.12:0/3629048516 wait complete. 2024-03-17T23:46:43.622 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling the dashboard module... 2024-03-17T23:46:43.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:43 smithi012 ceph-mon[31541]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:46:43.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:43 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3629048516' entity='client.admin' 2024-03-17T23:46:44.820 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:44 smithi012 ceph-mon[31541]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:46:44.821 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:44 smithi012 ceph-mon[31541]: from='mgr.14120 172.21.15.12:0/156503519' entity='mgr.x' 2024-03-17T23:46:44.821 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:44 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1435497885' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-17T23:46:45.126 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:44 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: ignoring --setuser ceph since I am not root 2024-03-17T23:46:45.126 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:44 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: ignoring --setgroup ceph since I am not root 2024-03-17T23:46:45.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.245+0000 7ff02b000700 1 Processor -- start 2024-03-17T23:46:45.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.246+0000 7ff02b000700 1 -- start start 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.246+0000 7ff02b000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 0x7ff024106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.246+0000 7ff02b000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff024107580 con 0x7ff024106b90 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.247+0000 7ff023e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 0x7ff024106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.247+0000 7ff023e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 0x7ff024106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52848/0 (socket says 172.21.15.12:52848) 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.247+0000 7ff023e00700 1 -- 172.21.15.12:0/3713045603 learned_addr learned my addr 172.21.15.12:0/3713045603 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.247+0000 7ff023e00700 1 -- 172.21.15.12:0/3713045603 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff024107d90 con 0x7ff024106b90 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.248+0000 7ff023e00700 1 --2- 172.21.15.12:0/3713045603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 0x7ff024106fb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff0140098a0 tx=0x7ff014009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b3b3fc910ef29f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.248+0000 7ff022a00700 1 -- 172.21.15.12:0/3713045603 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff014004160 con 0x7ff024106b90 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.248+0000 7ff022a00700 1 -- 172.21.15.12:0/3713045603 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff014004300 con 0x7ff024106b90 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.248+0000 7ff022a00700 1 -- 172.21.15.12:0/3713045603 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff014011b20 con 0x7ff024106b90 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.249+0000 7ff02b000700 1 -- 172.21.15.12:0/3713045603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 msgr2=0x7ff024106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.249+0000 7ff02b000700 1 --2- 172.21.15.12:0/3713045603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 0x7ff024106fb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff0140098a0 tx=0x7ff014009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.249+0000 7ff02b000700 1 -- 172.21.15.12:0/3713045603 shutdown_connections 2024-03-17T23:46:45.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.249+0000 7ff02b000700 1 --2- 172.21.15.12:0/3713045603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 0x7ff024106fb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:45.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.249+0000 7ff02b000700 1 -- 172.21.15.12:0/3713045603 >> 172.21.15.12:0/3713045603 conn(0x7ff024102110 msgr2=0x7ff024104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:45.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.249+0000 7ff02b000700 1 -- 172.21.15.12:0/3713045603 shutdown_connections 2024-03-17T23:46:45.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.249+0000 7ff02b000700 1 -- 172.21.15.12:0/3713045603 wait complete. 2024-03-17T23:46:45.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.250+0000 7ff02b000700 1 Processor -- start 2024-03-17T23:46:45.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.250+0000 7ff02b000700 1 -- start start 2024-03-17T23:46:45.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.251+0000 7ff02b000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 0x7ff02419bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:45.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.251+0000 7ff02b000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff02410b140 con 0x7ff024106b90 2024-03-17T23:46:45.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.251+0000 7ff023e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 0x7ff02419bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:45.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.251+0000 7ff023e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 0x7ff02419bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52864/0 (socket says 172.21.15.12:52864) 2024-03-17T23:46:45.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.251+0000 7ff023e00700 1 -- 172.21.15.12:0/1435497885 learned_addr learned my addr 172.21.15.12:0/1435497885 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:45.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.252+0000 7ff023e00700 1 -- 172.21.15.12:0/1435497885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff02419c4c0 con 0x7ff024106b90 2024-03-17T23:46:45.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.252+0000 7ff023e00700 1 --2- 172.21.15.12:0/1435497885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 0x7ff02419bf80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff0140117e0 tx=0x7ff014011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:45.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.252+0000 7ff020c00700 1 -- 172.21.15.12:0/1435497885 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff014003bf0 con 0x7ff024106b90 2024-03-17T23:46:45.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.252+0000 7ff020c00700 1 -- 172.21.15.12:0/1435497885 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff01401a910 con 0x7ff024106b90 2024-03-17T23:46:45.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.252+0000 7ff020c00700 1 -- 172.21.15.12:0/1435497885 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff014024970 con 0x7ff024106b90 2024-03-17T23:46:45.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.252+0000 7ff02b000700 1 -- 172.21.15.12:0/1435497885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff02419c750 con 0x7ff024106b90 2024-03-17T23:46:45.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.252+0000 7ff02b000700 1 -- 172.21.15.12:0/1435497885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff02419cbb0 con 0x7ff024106b90 2024-03-17T23:46:45.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.254+0000 7ff020c00700 1 -- 172.21.15.12:0/1435497885 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7ff014022020 con 0x7ff024106b90 2024-03-17T23:46:45.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.254+0000 7ff02b000700 1 -- 172.21.15.12:0/1435497885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff024063900 con 0x7ff024106b90 2024-03-17T23:46:45.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.254+0000 7ff020c00700 1 --2- 172.21.15.12:0/1435497885 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7ff008038f70 0x7ff00803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:45.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.254+0000 7ff020c00700 1 -- 172.21.15.12:0/1435497885 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff01404d270 con 0x7ff024106b90 2024-03-17T23:46:45.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.255+0000 7ff023400700 1 --2- 172.21.15.12:0/1435497885 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7ff008038f70 0x7ff00803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:45.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.255+0000 7ff023400700 1 --2- 172.21.15.12:0/1435497885 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7ff008038f70 0x7ff00803b430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff010006fd0 tx=0x7ff010006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:45.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.260+0000 7ff020c00700 1 -- 172.21.15.12:0/1435497885 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff01404cc00 con 0x7ff024106b90 2024-03-17T23:46:45.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.557+0000 7ff02b000700 1 -- 172.21.15.12:0/1435497885 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7ff02402dc10 con 0x7ff024106b90 2024-03-17T23:46:45.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.733+0000 7ff020c00700 1 -- 172.21.15.12:0/1435497885 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7ff01401ab70 con 0x7ff024106b90 2024-03-17T23:46:45.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.737+0000 7ff02b000700 1 -- 172.21.15.12:0/1435497885 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7ff008038f70 msgr2=0x7ff00803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:45.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.737+0000 7ff02b000700 1 --2- 172.21.15.12:0/1435497885 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7ff008038f70 0x7ff00803b430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff010006fd0 tx=0x7ff010006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:45.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.737+0000 7ff02b000700 1 -- 172.21.15.12:0/1435497885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 msgr2=0x7ff02419bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:45.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.737+0000 7ff02b000700 1 --2- 172.21.15.12:0/1435497885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 0x7ff02419bf80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff0140117e0 tx=0x7ff014011ba0 comp rx=0 tx=0).stop 2024-03-17T23:46:45.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.738+0000 7ff02b000700 1 -- 172.21.15.12:0/1435497885 shutdown_connections 2024-03-17T23:46:45.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.738+0000 7ff02b000700 1 --2- 172.21.15.12:0/1435497885 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7ff008038f70 0x7ff00803b430 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:45.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.738+0000 7ff02b000700 1 --2- 172.21.15.12:0/1435497885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024106b90 0x7ff02419bf80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:45.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.738+0000 7ff02b000700 1 -- 172.21.15.12:0/1435497885 >> 172.21.15.12:0/1435497885 conn(0x7ff024102110 msgr2=0x7ff024102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:45.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.739+0000 7ff02b000700 1 -- 172.21.15.12:0/1435497885 shutdown_connections 2024-03-17T23:46:45.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:44.739+0000 7ff02b000700 1 -- 172.21.15.12:0/1435497885 wait complete. 2024-03-17T23:46:45.735 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:45 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:45.609+0000 7fb9bad84200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-17T23:46:46.024 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:45 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:45.735+0000 7fb9bad84200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-17T23:46:46.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:45 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1435497885' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-17T23:46:46.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:45 smithi012 ceph-mon[31541]: mgrmap e9: x(active, since 25s) 2024-03-17T23:46:46.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:46 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2115121331' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-17T23:46:46.903 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:46 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:46.589+0000 7fb9bad84200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-17T23:46:46.903 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:46 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:46.716+0000 7fb9bad84200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-17T23:46:46.903 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:46 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:46.902+0000 7fb9bad84200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-17T23:46:47.040 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-17T23:46:47.046 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-17T23:46:47.046 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-17T23:46:47.047 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-17T23:46:47.047 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-17T23:46:47.047 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-17T23:46:47.047 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.079+0000 7fee5fc00700 1 Processor -- start 2024-03-17T23:46:47.047 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.079+0000 7fee5fc00700 1 -- start start 2024-03-17T23:46:47.047 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.080+0000 7fee5fc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 0x7fee5810a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:47.047 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.080+0000 7fee5fc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fee5810a7b0 con 0x7fee58109e50 2024-03-17T23:46:47.047 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.080+0000 7fee5d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 0x7fee5810a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:47.047 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.080+0000 7fee5d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 0x7fee5810a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52890/0 (socket says 172.21.15.12:52890) 2024-03-17T23:46:47.047 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.080+0000 7fee5d200700 1 -- 172.21.15.12:0/435790081 learned_addr learned my addr 172.21.15.12:0/435790081 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:47.047 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.081+0000 7fee5d200700 1 -- 172.21.15.12:0/435790081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5810b040 con 0x7fee58109e50 2024-03-17T23:46:47.047 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.081+0000 7fee5d200700 1 --2- 172.21.15.12:0/435790081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 0x7fee5810a270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0098a0 tx=0x7fee4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8542e2cd3f66ca7e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.081+0000 7fee57400700 1 -- 172.21.15.12:0/435790081 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee4c004160 con 0x7fee58109e50 2024-03-17T23:46:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.081+0000 7fee57400700 1 -- 172.21.15.12:0/435790081 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee4c004300 con 0x7fee58109e50 2024-03-17T23:46:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.081+0000 7fee57400700 1 -- 172.21.15.12:0/435790081 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee4c011b20 con 0x7fee58109e50 2024-03-17T23:46:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.082+0000 7fee5fc00700 1 -- 172.21.15.12:0/435790081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 msgr2=0x7fee5810a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.082+0000 7fee5fc00700 1 --2- 172.21.15.12:0/435790081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 0x7fee5810a270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0098a0 tx=0x7fee4c009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.082+0000 7fee5fc00700 1 -- 172.21.15.12:0/435790081 shutdown_connections 2024-03-17T23:46:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.082+0000 7fee5fc00700 1 --2- 172.21.15.12:0/435790081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 0x7fee5810a270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.082+0000 7fee5fc00700 1 -- 172.21.15.12:0/435790081 >> 172.21.15.12:0/435790081 conn(0x7fee58105580 msgr2=0x7fee58107970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.082+0000 7fee5fc00700 1 -- 172.21.15.12:0/435790081 shutdown_connections 2024-03-17T23:46:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.082+0000 7fee5fc00700 1 -- 172.21.15.12:0/435790081 wait complete. 2024-03-17T23:46:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.084+0000 7fee5fc00700 1 Processor -- start 2024-03-17T23:46:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.084+0000 7fee5fc00700 1 -- start start 2024-03-17T23:46:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.085+0000 7fee5fc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 0x7fee5819e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.085+0000 7fee5fc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fee5810e300 con 0x7fee58109e50 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.085+0000 7fee5d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 0x7fee5819e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.085+0000 7fee5d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 0x7fee5819e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52898/0 (socket says 172.21.15.12:52898) 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.085+0000 7fee5d200700 1 -- 172.21.15.12:0/2115121331 learned_addr learned my addr 172.21.15.12:0/2115121331 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.085+0000 7fee5d200700 1 -- 172.21.15.12:0/2115121331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5819e6c0 con 0x7fee58109e50 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.086+0000 7fee5d200700 1 --2- 172.21.15.12:0/2115121331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 0x7fee5819e180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0117e0 tx=0x7fee4c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.086+0000 7fee55600700 1 -- 172.21.15.12:0/2115121331 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee4c003bf0 con 0x7fee58109e50 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.086+0000 7fee55600700 1 -- 172.21.15.12:0/2115121331 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee4c01a910 con 0x7fee58109e50 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.086+0000 7fee5fc00700 1 -- 172.21.15.12:0/2115121331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee5819e950 con 0x7fee58109e50 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.086+0000 7fee55600700 1 -- 172.21.15.12:0/2115121331 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee4c024970 con 0x7fee58109e50 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.086+0000 7fee5fc00700 1 -- 172.21.15.12:0/2115121331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee5819edb0 con 0x7fee58109e50 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.087+0000 7fee5fc00700 1 -- 172.21.15.12:0/2115121331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee58063900 con 0x7fee58109e50 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.087+0000 7fee55600700 1 -- 172.21.15.12:0/2115121331 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7fee4c022020 con 0x7fee58109e50 2024-03-17T23:46:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.088+0000 7fee55600700 1 --2- 172.21.15.12:0/2115121331 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fee40038f70 0x7fee4003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.088+0000 7fee55600700 1 -- 172.21.15.12:0/2115121331 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fee4c04d6e0 con 0x7fee58109e50 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.088+0000 7fee57e00700 1 -- 172.21.15.12:0/2115121331 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fee40038f70 msgr2=0x7fee4003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2526360059 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.088+0000 7fee57e00700 1 --2- 172.21.15.12:0/2115121331 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fee40038f70 0x7fee4003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.097+0000 7fee55600700 1 -- 172.21.15.12:0/2115121331 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fee4c04d070 con 0x7fee58109e50 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.289+0000 7fee57e00700 1 -- 172.21.15.12:0/2115121331 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fee40038f70 msgr2=0x7fee4003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2526360059 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.289+0000 7fee57e00700 1 --2- 172.21.15.12:0/2115121331 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fee40038f70 0x7fee4003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.380+0000 7fee5fc00700 1 -- 172.21.15.12:0/2115121331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fee5802dc10 con 0x7fee58109e50 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.381+0000 7fee55600700 1 -- 172.21.15.12:0/2115121331 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fee4c017ca0 con 0x7fee58109e50 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.384+0000 7fee5fc00700 1 -- 172.21.15.12:0/2115121331 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fee40038f70 msgr2=0x7fee4003b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.384+0000 7fee5fc00700 1 --2- 172.21.15.12:0/2115121331 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fee40038f70 0x7fee4003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.384+0000 7fee5fc00700 1 -- 172.21.15.12:0/2115121331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 msgr2=0x7fee5819e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.384+0000 7fee5fc00700 1 --2- 172.21.15.12:0/2115121331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 0x7fee5819e180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0117e0 tx=0x7fee4c011ba0 comp rx=0 tx=0).stop 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.384+0000 7fee5fc00700 1 -- 172.21.15.12:0/2115121331 shutdown_connections 2024-03-17T23:46:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.384+0000 7fee5fc00700 1 --2- 172.21.15.12:0/2115121331 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7fee40038f70 0x7fee4003b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:47.051 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.384+0000 7fee5fc00700 1 --2- 172.21.15.12:0/2115121331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee58109e50 0x7fee5819e180 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:47.051 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.384+0000 7fee5fc00700 1 -- 172.21.15.12:0/2115121331 >> 172.21.15.12:0/2115121331 conn(0x7fee58105580 msgr2=0x7fee581071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:47.051 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.384+0000 7fee5fc00700 1 -- 172.21.15.12:0/2115121331 shutdown_connections 2024-03-17T23:46:47.051 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:46.384+0000 7fee5fc00700 1 -- 172.21.15.12:0/2115121331 wait complete. 2024-03-17T23:46:47.051 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for the mgr to restart... 2024-03-17T23:46:47.051 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr epoch 9... 2024-03-17T23:46:47.163 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:47.032+0000 7fb9bad84200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-17T23:46:47.164 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:47.162+0000 7fb9bad84200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-17T23:46:47.784 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:47.515+0000 7fb9bad84200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-17T23:46:47.784 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:47.667+0000 7fb9bad84200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-17T23:46:47.785 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:47.783+0000 7fb9bad84200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-17T23:46:48.136 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:47.912+0000 7fb9bad84200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-17T23:46:48.136 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:48 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:48.134+0000 7fb9bad84200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-17T23:46:49.919 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:49 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:49.450+0000 7fb9bad84200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-17T23:46:49.919 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:49 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:49.628+0000 7fb9bad84200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-17T23:46:50.423 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:50 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:50.055+0000 7fb9bad84200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-17T23:46:50.423 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:50 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:50.178+0000 7fb9bad84200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-17T23:46:50.701 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:50 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:50.445+0000 7fb9bad84200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:46:51.090 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:50 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:50.700+0000 7fb9bad84200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-17T23:46:51.090 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:51 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:51.088+0000 7fb9bad84200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:46:51.668 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:51 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:51.413+0000 7fb9bad84200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-17T23:46:52.169 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:51 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:51.729+0000 7fb9bad84200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-17T23:46:53.168 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:52 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:52.798+0000 7fb9bad84200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-17T23:46:53.872 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:53 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:53.521+0000 7fb9bad84200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-17T23:46:53.872 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:53 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:53.656+0000 7fb9bad84200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-17T23:46:54.132 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:54 smithi012 ceph-mon[31541]: Active manager daemon x restarted 2024-03-17T23:46:54.133 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:54 smithi012 ceph-mon[31541]: Activating manager daemon x 2024-03-17T23:46:54.133 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:54 smithi012 ceph-mon[31541]: osdmap e3: 0 total, 0 up, 0 in 2024-03-17T23:46:54.133 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:54 smithi012 ceph-mon[31541]: mgrmap e10: x(active, starting, since 0.00471765s) 2024-03-17T23:46:54.133 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:46:54.133 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-17T23:46:54.133 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-17T23:46:54.134 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-17T23:46:54.134 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-17T23:46:54.134 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:54 smithi012 ceph-mon[31541]: Manager daemon x is now available 2024-03-17T23:46:54.134 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:46:53 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:46:53.951+0000 7fb9bad84200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-17T23:46:55.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:46:55.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:46:55.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-17T23:46:55.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-17T23:46:55.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:55 smithi012 ceph-mon[31541]: [17/Mar/2024:23:46:54] ENGINE Bus STARTING 2024-03-17T23:46:55.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:55 smithi012 ceph-mon[31541]: mgrmap e11: x(active, since 1.00847s) 2024-03-17T23:46:55.692 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-17T23:46:55.693 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-17T23:46:55.694 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-17T23:46:55.694 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-17T23:46:55.694 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.653+0000 7f334dc00700 1 Processor -- start 2024-03-17T23:46:55.694 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.653+0000 7f334dc00700 1 -- start start 2024-03-17T23:46:55.694 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.653+0000 7f334dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 0x7f3348106950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:55.694 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.653+0000 7f334dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f334807beb0 con 0x7f3348106530 2024-03-17T23:46:55.694 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.653+0000 7f3346a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 0x7f3348106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:55.694 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.654+0000 7f3346a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 0x7f3348106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52908/0 (socket says 172.21.15.12:52908) 2024-03-17T23:46:55.694 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.654+0000 7f3346a00700 1 -- 172.21.15.12:0/4217210264 learned_addr learned my addr 172.21.15.12:0/4217210264 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:55.694 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.654+0000 7f3346a00700 1 -- 172.21.15.12:0/4217210264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3348106e90 con 0x7f3348106530 2024-03-17T23:46:55.694 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.654+0000 7f3346a00700 1 --2- 172.21.15.12:0/4217210264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 0x7f3348106950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f33380098a0 tx=0x7f3338009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3e6940efea733a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:55.695 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.655+0000 7f3345600700 1 -- 172.21.15.12:0/4217210264 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3338004160 con 0x7f3348106530 2024-03-17T23:46:55.695 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.655+0000 7f3345600700 1 -- 172.21.15.12:0/4217210264 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3338004300 con 0x7f3348106530 2024-03-17T23:46:55.695 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.655+0000 7f3345600700 1 -- 172.21.15.12:0/4217210264 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3338011b20 con 0x7f3348106530 2024-03-17T23:46:55.695 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.655+0000 7f334dc00700 1 -- 172.21.15.12:0/4217210264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 msgr2=0x7f3348106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:55.695 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.656+0000 7f334dc00700 1 --2- 172.21.15.12:0/4217210264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 0x7f3348106950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f33380098a0 tx=0x7f3338009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.695 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.656+0000 7f334dc00700 1 -- 172.21.15.12:0/4217210264 shutdown_connections 2024-03-17T23:46:55.695 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.656+0000 7f334dc00700 1 --2- 172.21.15.12:0/4217210264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 0x7f3348106950 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.695 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.656+0000 7f334dc00700 1 -- 172.21.15.12:0/4217210264 >> 172.21.15.12:0/4217210264 conn(0x7f33481021b0 msgr2=0x7f33481045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:55.695 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.656+0000 7f334dc00700 1 -- 172.21.15.12:0/4217210264 shutdown_connections 2024-03-17T23:46:55.695 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.656+0000 7f334dc00700 1 -- 172.21.15.12:0/4217210264 wait complete. 2024-03-17T23:46:55.695 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.657+0000 7f334dc00700 1 Processor -- start 2024-03-17T23:46:55.696 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.658+0000 7f334dc00700 1 -- start start 2024-03-17T23:46:55.696 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.658+0000 7f334dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 0x7f334819e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:55.696 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.658+0000 7f334dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3348107f60 con 0x7f3348106530 2024-03-17T23:46:55.696 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.658+0000 7f3346a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 0x7f334819e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:55.696 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.658+0000 7f3346a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 0x7f334819e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52924/0 (socket says 172.21.15.12:52924) 2024-03-17T23:46:55.696 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.658+0000 7f3346a00700 1 -- 172.21.15.12:0/2634226437 learned_addr learned my addr 172.21.15.12:0/2634226437 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:55.696 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.659+0000 7f3346a00700 1 -- 172.21.15.12:0/2634226437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f334819e5e0 con 0x7f3348106530 2024-03-17T23:46:55.696 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.659+0000 7f3346a00700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 0x7f334819e0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f33380117e0 tx=0x7f3338011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:55.696 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.660+0000 7f333f400700 1 -- 172.21.15.12:0/2634226437 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3338003bf0 con 0x7f3348106530 2024-03-17T23:46:55.696 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.660+0000 7f333f400700 1 -- 172.21.15.12:0/2634226437 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f333801a910 con 0x7f3348106530 2024-03-17T23:46:55.696 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.660+0000 7f334dc00700 1 -- 172.21.15.12:0/2634226437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f334819e870 con 0x7f3348106530 2024-03-17T23:46:55.697 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.660+0000 7f333f400700 1 -- 172.21.15.12:0/2634226437 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3338024970 con 0x7f3348106530 2024-03-17T23:46:55.697 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.660+0000 7f334dc00700 1 -- 172.21.15.12:0/2634226437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f334819ecd0 con 0x7f3348106530 2024-03-17T23:46:55.697 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.661+0000 7f333f400700 1 -- 172.21.15.12:0/2634226437 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f3338022020 con 0x7f3348106530 2024-03-17T23:46:55.697 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.662+0000 7f333f400700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 0x7f332c03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:55.697 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.662+0000 7f333f400700 1 -- 172.21.15.12:0/2634226437 --> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f332c03bb50 con 0x7f332c038f80 2024-03-17T23:46:55.697 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.662+0000 7f333f400700 1 -- 172.21.15.12:0/2634226437 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f333804d1c0 con 0x7f3348106530 2024-03-17T23:46:55.697 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.662+0000 7f3346000700 1 -- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 msgr2=0x7f332c03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2526360059 2024-03-17T23:46:55.697 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.662+0000 7f3346000700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 0x7f332c03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-17T23:46:55.697 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.863+0000 7f3346000700 1 -- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 msgr2=0x7f332c03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2526360059 2024-03-17T23:46:55.697 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:47.863+0000 7f3346000700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 0x7f332c03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-17T23:46:55.697 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:48.264+0000 7f3346000700 1 -- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 msgr2=0x7f332c03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2526360059 2024-03-17T23:46:55.698 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:48.264+0000 7f3346000700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 0x7f332c03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-17T23:46:55.698 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:49.065+0000 7f3346000700 1 -- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 msgr2=0x7f332c03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2526360059 2024-03-17T23:46:55.698 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:49.065+0000 7f3346000700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 0x7f332c03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-17T23:46:55.698 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:50.667+0000 7f3346000700 1 -- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 msgr2=0x7f332c03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2526360059 2024-03-17T23:46:55.698 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:50.667+0000 7f3346000700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 0x7f332c03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-17T23:46:55.698 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:53.870+0000 7f3346000700 1 -- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 msgr2=0x7f332c03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2526360059 2024-03-17T23:46:55.698 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:53.871+0000 7f3346000700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 0x7f332c03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-17T23:46:55.698 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:53.959+0000 7f333f400700 1 -- 172.21.15.12:0/2634226437 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7f333801a430 con 0x7f3348106530 2024-03-17T23:46:55.698 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:53.960+0000 7f333f400700 1 -- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 msgr2=0x7f332c03b440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-17T23:46:55.698 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:53.960+0000 7f333f400700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 0x7f332c03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.698 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.961+0000 7f333f400700 1 -- 172.21.15.12:0/2634226437 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f333804efd0 con 0x7f3348106530 2024-03-17T23:46:55.699 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.961+0000 7f333f400700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f332c03ca30 0x7f332c03ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:55.699 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.961+0000 7f333f400700 1 -- 172.21.15.12:0/2634226437 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f33380068a0 con 0x7f332c03ca30 2024-03-17T23:46:55.699 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.963+0000 7f3346000700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f332c03ca30 0x7f332c03ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:55.699 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.964+0000 7f3346000700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f332c03ca30 0x7f332c03ee20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3334003a10 tx=0x7f3334009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:55.699 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.965+0000 7f333f400700 1 -- 172.21.15.12:0/2634226437 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f33380068a0 con 0x7f332c03ca30 2024-03-17T23:46:55.699 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.973+0000 7f334dc00700 1 -- 172.21.15.12:0/2634226437 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f3348063900 con 0x7f332c03ca30 2024-03-17T23:46:55.699 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.973+0000 7f333f400700 1 -- 172.21.15.12:0/2634226437 <== mgr.14152 v2:172.21.15.12:6800/3179836366 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f3348063900 con 0x7f332c03ca30 2024-03-17T23:46:55.699 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.973+0000 7f334dc00700 1 -- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f332c03ca30 msgr2=0x7f332c03ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:55.699 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.973+0000 7f334dc00700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f332c03ca30 0x7f332c03ee20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3334003a10 tx=0x7f3334009270 comp rx=0 tx=0).stop 2024-03-17T23:46:55.699 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.974+0000 7f334dc00700 1 -- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 msgr2=0x7f334819e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:55.699 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.974+0000 7f334dc00700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 0x7f334819e0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f33380117e0 tx=0x7f3338011c10 comp rx=0 tx=0).stop 2024-03-17T23:46:55.700 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.974+0000 7f334dc00700 1 -- 172.21.15.12:0/2634226437 shutdown_connections 2024-03-17T23:46:55.700 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.974+0000 7f334dc00700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f332c03ca30 0x7f332c03ee20 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.700 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.974+0000 7f334dc00700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3348106530 0x7f334819e0a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.700 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.974+0000 7f334dc00700 1 --2- 172.21.15.12:0/2634226437 >> [v2:172.21.15.12:6800/2526360059,v1:172.21.15.12:6801/2526360059] conn(0x7f332c038f80 0x7f332c03b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.700 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.974+0000 7f334dc00700 1 -- 172.21.15.12:0/2634226437 >> 172.21.15.12:0/2634226437 conn(0x7f33481021b0 msgr2=0x7f3348194a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:55.700 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.974+0000 7f334dc00700 1 -- 172.21.15.12:0/2634226437 shutdown_connections 2024-03-17T23:46:55.700 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:54.974+0000 7f334dc00700 1 -- 172.21.15.12:0/2634226437 wait complete. 2024-03-17T23:46:55.700 INFO:teuthology.orchestra.run.smithi012.stdout:mgr epoch 9 is available 2024-03-17T23:46:55.700 INFO:teuthology.orchestra.run.smithi012.stdout:Generating a dashboard self-signed certificate... 2024-03-17T23:46:56.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:56 smithi012 ceph-mon[31541]: [17/Mar/2024:23:46:54] ENGINE Serving on https://172.21.15.12:7150 2024-03-17T23:46:56.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:56 smithi012 ceph-mon[31541]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-17T23:46:56.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:56 smithi012 ceph-mon[31541]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-17T23:46:56.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:56 smithi012 ceph-mon[31541]: [17/Mar/2024:23:46:55] ENGINE Serving on http://172.21.15.12:8765 2024-03-17T23:46:56.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:56 smithi012 ceph-mon[31541]: [17/Mar/2024:23:46:55] ENGINE Bus STARTED 2024-03-17T23:46:57.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:57 smithi012 ceph-mon[31541]: mgrmap e12: x(active, since 2s) 2024-03-17T23:46:57.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:46:58.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-17T23:46:58.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.855+0000 7fe436400700 1 Processor -- start 2024-03-17T23:46:58.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.855+0000 7fe436400700 1 -- start start 2024-03-17T23:46:58.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.855+0000 7fe436400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 0x7fe430106310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:58.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.855+0000 7fe436400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe4301068e0 con 0x7fe430105ef0 2024-03-17T23:46:58.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.856+0000 7fe42f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 0x7fe430106310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:58.005 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.856+0000 7fe42f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 0x7fe430106310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44478/0 (socket says 172.21.15.12:44478) 2024-03-17T23:46:58.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.856+0000 7fe42f400700 1 -- 172.21.15.12:0/1971767965 learned_addr learned my addr 172.21.15.12:0/1971767965 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:58.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.857+0000 7fe42f400700 1 -- 172.21.15.12:0/1971767965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe430106a60 con 0x7fe430105ef0 2024-03-17T23:46:58.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.857+0000 7fe42f400700 1 --2- 172.21.15.12:0/1971767965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 0x7fe430106310 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe4200098a0 tx=0x7fe420009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e0f3d78dd08d9f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:58.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.857+0000 7fe42e000700 1 -- 172.21.15.12:0/1971767965 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe420004160 con 0x7fe430105ef0 2024-03-17T23:46:58.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.857+0000 7fe42e000700 1 -- 172.21.15.12:0/1971767965 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe420004300 con 0x7fe430105ef0 2024-03-17T23:46:58.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.857+0000 7fe42e000700 1 -- 172.21.15.12:0/1971767965 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe420011b20 con 0x7fe430105ef0 2024-03-17T23:46:58.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.858+0000 7fe436400700 1 -- 172.21.15.12:0/1971767965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 msgr2=0x7fe430106310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:58.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.858+0000 7fe436400700 1 --2- 172.21.15.12:0/1971767965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 0x7fe430106310 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe4200098a0 tx=0x7fe420009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:58.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.858+0000 7fe436400700 1 -- 172.21.15.12:0/1971767965 shutdown_connections 2024-03-17T23:46:58.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.858+0000 7fe436400700 1 --2- 172.21.15.12:0/1971767965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 0x7fe430106310 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:58.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.858+0000 7fe436400700 1 -- 172.21.15.12:0/1971767965 >> 172.21.15.12:0/1971767965 conn(0x7fe4301018d0 msgr2=0x7fe430103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:58.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.858+0000 7fe436400700 1 -- 172.21.15.12:0/1971767965 shutdown_connections 2024-03-17T23:46:58.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.859+0000 7fe436400700 1 -- 172.21.15.12:0/1971767965 wait complete. 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.860+0000 7fe436400700 1 Processor -- start 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.860+0000 7fe436400700 1 -- start start 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.860+0000 7fe436400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 0x7fe4301087f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.860+0000 7fe436400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe43010c1a0 con 0x7fe430105ef0 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.860+0000 7fe42f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 0x7fe4301087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.860+0000 7fe42f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 0x7fe4301087f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44490/0 (socket says 172.21.15.12:44490) 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.860+0000 7fe42f400700 1 -- 172.21.15.12:0/2477316029 learned_addr learned my addr 172.21.15.12:0/2477316029 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.861+0000 7fe42f400700 1 -- 172.21.15.12:0/2477316029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe430108d30 con 0x7fe430105ef0 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.861+0000 7fe42f400700 1 --2- 172.21.15.12:0/2477316029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 0x7fe4301087f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe4200117e0 tx=0x7fe420011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.862+0000 7fe427e00700 1 -- 172.21.15.12:0/2477316029 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe420003bf0 con 0x7fe430105ef0 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.862+0000 7fe427e00700 1 -- 172.21.15.12:0/2477316029 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42001a910 con 0x7fe430105ef0 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.862+0000 7fe436400700 1 -- 172.21.15.12:0/2477316029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe430106f20 con 0x7fe430105ef0 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.862+0000 7fe427e00700 1 -- 172.21.15.12:0/2477316029 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe420024970 con 0x7fe430105ef0 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.862+0000 7fe436400700 1 -- 172.21.15.12:0/2477316029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4301073e0 con 0x7fe430105ef0 2024-03-17T23:46:58.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.863+0000 7fe427e00700 1 -- 172.21.15.12:0/2477316029 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fe420022020 con 0x7fe430105ef0 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.863+0000 7fe436400700 1 -- 172.21.15.12:0/2477316029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe430063900 con 0x7fe430105ef0 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.863+0000 7fe427e00700 1 --2- 172.21.15.12:0/2477316029 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe414038fd0 0x7fe41403b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.864+0000 7fe427e00700 1 -- 172.21.15.12:0/2477316029 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe42004d3b0 con 0x7fe430105ef0 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.864+0000 7fe42ea00700 1 --2- 172.21.15.12:0/2477316029 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe414038fd0 0x7fe41403b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.865+0000 7fe42ea00700 1 --2- 172.21.15.12:0/2477316029 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe414038fd0 0x7fe41403b490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe41c006fd0 tx=0x7fe41c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:56.870+0000 7fe427e00700 1 -- 172.21.15.12:0/2477316029 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe42002d910 con 0x7fe430105ef0 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:57.105+0000 7fe436400700 1 -- 172.21.15.12:0/2477316029 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fe430107c10 con 0x7fe414038fd0 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:57.325+0000 7fe427e00700 1 -- 172.21.15.12:0/2477316029 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fe430107c10 con 0x7fe414038fd0 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:57.328+0000 7fe436400700 1 -- 172.21.15.12:0/2477316029 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe414038fd0 msgr2=0x7fe41403b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:57.328+0000 7fe436400700 1 --2- 172.21.15.12:0/2477316029 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe414038fd0 0x7fe41403b490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe41c006fd0 tx=0x7fe41c006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:57.328+0000 7fe436400700 1 -- 172.21.15.12:0/2477316029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 msgr2=0x7fe4301087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:57.328+0000 7fe436400700 1 --2- 172.21.15.12:0/2477316029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 0x7fe4301087f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe4200117e0 tx=0x7fe420011ba0 comp rx=0 tx=0).stop 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:57.328+0000 7fe436400700 1 -- 172.21.15.12:0/2477316029 shutdown_connections 2024-03-17T23:46:58.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:57.328+0000 7fe436400700 1 --2- 172.21.15.12:0/2477316029 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe414038fd0 0x7fe41403b490 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:58.009 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:57.328+0000 7fe436400700 1 --2- 172.21.15.12:0/2477316029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe430105ef0 0x7fe4301087f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:58.009 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:57.328+0000 7fe436400700 1 -- 172.21.15.12:0/2477316029 >> 172.21.15.12:0/2477316029 conn(0x7fe4301018d0 msgr2=0x7fe4301023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:58.009 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:57.329+0000 7fe436400700 1 -- 172.21.15.12:0/2477316029 shutdown_connections 2024-03-17T23:46:58.009 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:57.329+0000 7fe436400700 1 -- 172.21.15.12:0/2477316029 wait complete. 2024-03-17T23:46:58.009 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial admin user... 2024-03-17T23:46:58.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:58 smithi012 ceph-mon[31541]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:46:58.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:58 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:46:58.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:58 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:46:59.805 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$iPvJXGrpxdFsqUEUt348zeWWIxwkfGf5WB7Zf7Gn.rz.wxmMTUWo2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710719219, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-17T23:46:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.592+0000 7f8406e00700 1 Processor -- start 2024-03-17T23:46:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.592+0000 7f8406e00700 1 -- start start 2024-03-17T23:46:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.592+0000 7f8406e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 0x7f8400106950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.592+0000 7f8406e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f84000759c0 con 0x7f8400106530 2024-03-17T23:46:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.593+0000 7f83ffe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 0x7f8400106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.593+0000 7f83ffe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 0x7f8400106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44506/0 (socket says 172.21.15.12:44506) 2024-03-17T23:46:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.593+0000 7f83ffe00700 1 -- 172.21.15.12:0/1300757642 learned_addr learned my addr 172.21.15.12:0/1300757642 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.593+0000 7f83ffe00700 1 -- 172.21.15.12:0/1300757642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8400106e90 con 0x7f8400106530 2024-03-17T23:46:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.594+0000 7f83ffe00700 1 --2- 172.21.15.12:0/1300757642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 0x7f8400106950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f83f00098a0 tx=0x7f83f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb349a39e84a3bd server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:59.806 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.594+0000 7f83ff400700 1 -- 172.21.15.12:0/1300757642 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83f0004160 con 0x7f8400106530 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.594+0000 7f83ff400700 1 -- 172.21.15.12:0/1300757642 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83f0004300 con 0x7f8400106530 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.594+0000 7f83ff400700 1 -- 172.21.15.12:0/1300757642 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83f0011b20 con 0x7f8400106530 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.595+0000 7f8406e00700 1 -- 172.21.15.12:0/1300757642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 msgr2=0x7f8400106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.595+0000 7f8406e00700 1 --2- 172.21.15.12:0/1300757642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 0x7f8400106950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f83f00098a0 tx=0x7f83f0009bb0 comp rx=0 tx=0).stop 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.595+0000 7f8406e00700 1 -- 172.21.15.12:0/1300757642 shutdown_connections 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.595+0000 7f8406e00700 1 --2- 172.21.15.12:0/1300757642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 0x7f8400106950 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.595+0000 7f8406e00700 1 -- 172.21.15.12:0/1300757642 >> 172.21.15.12:0/1300757642 conn(0x7f8400102130 msgr2=0x7f8400104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.595+0000 7f8406e00700 1 -- 172.21.15.12:0/1300757642 shutdown_connections 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.595+0000 7f8406e00700 1 -- 172.21.15.12:0/1300757642 wait complete. 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.596+0000 7f8406e00700 1 Processor -- start 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.596+0000 7f8406e00700 1 -- start start 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.596+0000 7f8406e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 0x7f840019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.597+0000 7f8406e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8400107f60 con 0x7f8400106530 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.597+0000 7f83ffe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 0x7f840019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:59.807 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.597+0000 7f83ffe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 0x7f840019dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44510/0 (socket says 172.21.15.12:44510) 2024-03-17T23:46:59.808 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.597+0000 7f83ffe00700 1 -- 172.21.15.12:0/1818879678 learned_addr learned my addr 172.21.15.12:0/1818879678 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:46:59.808 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.597+0000 7f83ffe00700 1 -- 172.21.15.12:0/1818879678 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840019e2e0 con 0x7f8400106530 2024-03-17T23:46:59.808 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.598+0000 7f83ffe00700 1 --2- 172.21.15.12:0/1818879678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 0x7f840019dda0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f83f00117e0 tx=0x7f83f0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:59.808 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.598+0000 7f83fd600700 1 -- 172.21.15.12:0/1818879678 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83f0003ba0 con 0x7f8400106530 2024-03-17T23:46:59.808 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.598+0000 7f83fd600700 1 -- 172.21.15.12:0/1818879678 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83f001a910 con 0x7f8400106530 2024-03-17T23:46:59.808 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.598+0000 7f8406e00700 1 -- 172.21.15.12:0/1818879678 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f840019e570 con 0x7f8400106530 2024-03-17T23:46:59.808 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.598+0000 7f83fd600700 1 -- 172.21.15.12:0/1818879678 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83f00248f0 con 0x7f8400106530 2024-03-17T23:46:59.808 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.599+0000 7f8406e00700 1 -- 172.21.15.12:0/1818879678 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84001a1220 con 0x7f8400106530 2024-03-17T23:46:59.808 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.600+0000 7f83fd600700 1 -- 172.21.15.12:0/1818879678 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f83f0022020 con 0x7f8400106530 2024-03-17T23:46:59.808 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.600+0000 7f83fd600700 1 --2- 172.21.15.12:0/1818879678 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83e4038fd0 0x7f83e403b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:59.808 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.600+0000 7f83fd600700 1 -- 172.21.15.12:0/1818879678 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f83f0052660 con 0x7f8400106530 2024-03-17T23:46:59.808 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.600+0000 7f8406e00700 1 -- 172.21.15.12:0/1818879678 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8400063900 con 0x7f8400106530 2024-03-17T23:46:59.808 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.601+0000 7f83f7e00700 1 --2- 172.21.15.12:0/1818879678 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83e4038fd0 0x7f83e403b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:59.809 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.602+0000 7f83f7e00700 1 --2- 172.21.15.12:0/1818879678 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83e4038fd0 0x7f83e403b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f83e8006fd0 tx=0x7f83e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:59.809 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.610+0000 7f83fd600700 1 -- 172.21.15.12:0/1818879678 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f83f0022310 con 0x7f8400106530 2024-03-17T23:46:59.809 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:58.862+0000 7f8406e00700 1 -- 172.21.15.12:0/1818879678 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f8400197860 con 0x7f83e4038fd0 2024-03-17T23:46:59.809 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:59.122+0000 7f83fd600700 1 -- 172.21.15.12:0/1818879678 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f8400197860 con 0x7f83e4038fd0 2024-03-17T23:46:59.809 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:59.126+0000 7f8406e00700 1 -- 172.21.15.12:0/1818879678 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83e4038fd0 msgr2=0x7f83e403b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:59.809 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:59.126+0000 7f8406e00700 1 --2- 172.21.15.12:0/1818879678 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83e4038fd0 0x7f83e403b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f83e8006fd0 tx=0x7f83e8006e40 comp rx=0 tx=0).stop 2024-03-17T23:46:59.809 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:59.126+0000 7f8406e00700 1 -- 172.21.15.12:0/1818879678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 msgr2=0x7f840019dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:59.809 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:59.126+0000 7f8406e00700 1 --2- 172.21.15.12:0/1818879678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 0x7f840019dda0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f83f00117e0 tx=0x7f83f0011ba0 comp rx=0 tx=0).stop 2024-03-17T23:46:59.809 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:59.126+0000 7f8406e00700 1 -- 172.21.15.12:0/1818879678 shutdown_connections 2024-03-17T23:46:59.809 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:59.126+0000 7f8406e00700 1 --2- 172.21.15.12:0/1818879678 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83e4038fd0 0x7f83e403b490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:59.809 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:59.126+0000 7f8406e00700 1 --2- 172.21.15.12:0/1818879678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8400106530 0x7f840019dda0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:59.809 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:59.126+0000 7f8406e00700 1 -- 172.21.15.12:0/1818879678 >> 172.21.15.12:0/1818879678 conn(0x7f8400102130 msgr2=0x7f84001949a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:59.809 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:59.126+0000 7f8406e00700 1 -- 172.21.15.12:0/1818879678 shutdown_connections 2024-03-17T23:46:59.810 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:46:59.127+0000 7f8406e00700 1 -- 172.21.15.12:0/1818879678 wait complete. 2024-03-17T23:46:59.810 INFO:teuthology.orchestra.run.smithi012.stdout:Fetching dashboard port number... 2024-03-17T23:46:59.957 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:46:59.957 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:59 smithi012 ceph-mon[31541]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:46:59.957 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:46:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:00.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:00 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/716786438' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-17T23:47:01.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:01 smithi012 ceph-mon[31541]: mgrmap e13: x(active, since 6s) 2024-03-17T23:47:02.127 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 8443 2024-03-17T23:47:02.127 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.388+0000 7fbeb1e00700 1 Processor -- start 2024-03-17T23:47:02.127 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.388+0000 7fbeb1e00700 1 -- start start 2024-03-17T23:47:02.127 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.388+0000 7fbeb1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 0x7fbeac10aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.128 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.388+0000 7fbeb1e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbeac075a40 con 0x7fbeac1086c0 2024-03-17T23:47:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.389+0000 7fbeaaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 0x7fbeac10aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.389+0000 7fbeaaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 0x7fbeac10aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44524/0 (socket says 172.21.15.12:44524) 2024-03-17T23:47:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.389+0000 7fbeaaa00700 1 -- 172.21.15.12:0/514662047 learned_addr learned my addr 172.21.15.12:0/514662047 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:47:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.389+0000 7fbeaaa00700 1 -- 172.21.15.12:0/514662047 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeac10aff0 con 0x7fbeac1086c0 2024-03-17T23:47:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.389+0000 7fbeaaa00700 1 --2- 172.21.15.12:0/514662047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 0x7fbeac10aab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0098a0 tx=0x7fbe9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d6a31c43868d1f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.390+0000 7fbea9600700 1 -- 172.21.15.12:0/514662047 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe9c004160 con 0x7fbeac1086c0 2024-03-17T23:47:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.390+0000 7fbea9600700 1 -- 172.21.15.12:0/514662047 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe9c004300 con 0x7fbeac1086c0 2024-03-17T23:47:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.390+0000 7fbea9600700 1 -- 172.21.15.12:0/514662047 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe9c011b20 con 0x7fbeac1086c0 2024-03-17T23:47:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.391+0000 7fbeb1e00700 1 -- 172.21.15.12:0/514662047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 msgr2=0x7fbeac10aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.391+0000 7fbeb1e00700 1 --2- 172.21.15.12:0/514662047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 0x7fbeac10aab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0098a0 tx=0x7fbe9c009bb0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.391+0000 7fbeb1e00700 1 -- 172.21.15.12:0/514662047 shutdown_connections 2024-03-17T23:47:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.391+0000 7fbeb1e00700 1 --2- 172.21.15.12:0/514662047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 0x7fbeac10aab0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.391+0000 7fbeb1e00700 1 -- 172.21.15.12:0/514662047 >> 172.21.15.12:0/514662047 conn(0x7fbeac102130 msgr2=0x7fbeac104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:02.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.391+0000 7fbeb1e00700 1 -- 172.21.15.12:0/514662047 shutdown_connections 2024-03-17T23:47:02.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.391+0000 7fbeb1e00700 1 -- 172.21.15.12:0/514662047 wait complete. 2024-03-17T23:47:02.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.392+0000 7fbeb1e00700 1 Processor -- start 2024-03-17T23:47:02.133 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.392+0000 7fbeb1e00700 1 -- start start 2024-03-17T23:47:02.133 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.392+0000 7fbeb1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 0x7fbeac19bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.133 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.392+0000 7fbeb1e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbeac10c060 con 0x7fbeac1086c0 2024-03-17T23:47:02.133 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.393+0000 7fbeaaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 0x7fbeac19bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.393+0000 7fbeaaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 0x7fbeac19bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44526/0 (socket says 172.21.15.12:44526) 2024-03-17T23:47:02.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.393+0000 7fbeaaa00700 1 -- 172.21.15.12:0/716786438 learned_addr learned my addr 172.21.15.12:0/716786438 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:47:02.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.393+0000 7fbeaaa00700 1 -- 172.21.15.12:0/716786438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeac19c0c0 con 0x7fbeac1086c0 2024-03-17T23:47:02.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.394+0000 7fbeaaa00700 1 --2- 172.21.15.12:0/716786438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 0x7fbeac19bb80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0117e0 tx=0x7fbe9c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:02.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.394+0000 7fbeb0a00700 1 -- 172.21.15.12:0/716786438 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe9c003ba0 con 0x7fbeac1086c0 2024-03-17T23:47:02.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.394+0000 7fbeb0a00700 1 -- 172.21.15.12:0/716786438 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe9c01a910 con 0x7fbeac1086c0 2024-03-17T23:47:02.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.394+0000 7fbeb1e00700 1 -- 172.21.15.12:0/716786438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeac19c350 con 0x7fbeac1086c0 2024-03-17T23:47:02.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.394+0000 7fbeb0a00700 1 -- 172.21.15.12:0/716786438 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe9c0248f0 con 0x7fbeac1086c0 2024-03-17T23:47:02.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.394+0000 7fbeb1e00700 1 -- 172.21.15.12:0/716786438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeac19c7b0 con 0x7fbeac1086c0 2024-03-17T23:47:02.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.395+0000 7fbeb0a00700 1 -- 172.21.15.12:0/716786438 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fbe9c022020 con 0x7fbeac1086c0 2024-03-17T23:47:02.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.395+0000 7fbeb1e00700 1 -- 172.21.15.12:0/716786438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeac063900 con 0x7fbeac1086c0 2024-03-17T23:47:02.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.396+0000 7fbeb0a00700 1 --2- 172.21.15.12:0/716786438 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe90038fd0 0x7fbe9003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.396+0000 7fbeb0a00700 1 -- 172.21.15.12:0/716786438 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbe9c04e810 con 0x7fbeac1086c0 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.396+0000 7fbeaa000700 1 --2- 172.21.15.12:0/716786438 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe90038fd0 0x7fbe9003b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.397+0000 7fbeaa000700 1 --2- 172.21.15.12:0/716786438 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe90038fd0 0x7fbe9003b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbe98006fd0 tx=0x7fbe98006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.402+0000 7fbeb0a00700 1 -- 172.21.15.12:0/716786438 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbe9c00b3e0 con 0x7fbeac1086c0 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.634+0000 7fbeb1e00700 1 -- 172.21.15.12:0/716786438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fbeac19f180 con 0x7fbeac1086c0 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.635+0000 7fbeb0a00700 1 -- 172.21.15.12:0/716786438 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fbe9c0172b0 con 0x7fbeac1086c0 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.638+0000 7fbeb1e00700 1 -- 172.21.15.12:0/716786438 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe90038fd0 msgr2=0x7fbe9003b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.638+0000 7fbeb1e00700 1 --2- 172.21.15.12:0/716786438 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe90038fd0 0x7fbe9003b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbe98006fd0 tx=0x7fbe98006e40 comp rx=0 tx=0).stop 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.638+0000 7fbeb1e00700 1 -- 172.21.15.12:0/716786438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 msgr2=0x7fbeac19bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.638+0000 7fbeb1e00700 1 --2- 172.21.15.12:0/716786438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 0x7fbeac19bb80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0117e0 tx=0x7fbe9c011ba0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.638+0000 7fbeb1e00700 1 -- 172.21.15.12:0/716786438 shutdown_connections 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.638+0000 7fbeb1e00700 1 --2- 172.21.15.12:0/716786438 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe90038fd0 0x7fbe9003b490 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.638+0000 7fbeb1e00700 1 --2- 172.21.15.12:0/716786438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeac1086c0 0x7fbeac19bb80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.638+0000 7fbeb1e00700 1 -- 172.21.15.12:0/716786438 >> 172.21.15.12:0/716786438 conn(0x7fbeac102130 msgr2=0x7fbeac102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:02.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.639+0000 7fbeb1e00700 1 -- 172.21.15.12:0/716786438 shutdown_connections 2024-03-17T23:47:02.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:00.639+0000 7fbeb1e00700 1 -- 172.21.15.12:0/716786438 wait complete. 2024-03-17T23:47:02.159 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-17T23:47:02.159 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-03-17T23:47:02.179 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-17T23:47:02.179 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-17T23:47:02.179 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-03-17T23:47:02.179 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-17T23:47:02.182 INFO:teuthology.orchestra.run.smithi012.stdout:Ceph Dashboard is now available at: 2024-03-17T23:47:02.183 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:47:02.183 INFO:teuthology.orchestra.run.smithi012.stdout: URL: https://smithi012.front.sepia.ceph.com:8443/ 2024-03-17T23:47:02.183 INFO:teuthology.orchestra.run.smithi012.stdout: User: admin 2024-03-17T23:47:02.183 INFO:teuthology.orchestra.run.smithi012.stdout: Password: 3d5hdq8cr5 2024-03-17T23:47:02.183 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:47:02.183 INFO:teuthology.orchestra.run.smithi012.stdout:Saving cluster configuration to /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config directory 2024-03-17T23:47:02.184 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling autotune for osd_memory_target 2024-03-17T23:47:03.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.830+0000 7fbeb8600700 1 Processor -- start 2024-03-17T23:47:03.795 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.831+0000 7fbeb8600700 1 -- start start 2024-03-17T23:47:03.795 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.831+0000 7fbeb8600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 0x7fbeb007b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.795 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.831+0000 7fbeb8600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb007b7a0 con 0x7fbeb007cd70 2024-03-17T23:47:03.795 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.831+0000 7fbeb5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 0x7fbeb007b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.795 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.832+0000 7fbeb5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 0x7fbeb007b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44540/0 (socket says 172.21.15.12:44540) 2024-03-17T23:47:03.795 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.832+0000 7fbeb5c00700 1 -- 172.21.15.12:0/4100376042 learned_addr learned my addr 172.21.15.12:0/4100376042 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:47:03.795 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.832+0000 7fbeb5c00700 1 -- 172.21.15.12:0/4100376042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb007b920 con 0x7fbeb007cd70 2024-03-17T23:47:03.795 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.832+0000 7fbeb5c00700 1 --2- 172.21.15.12:0/4100376042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 0x7fbeb007b1d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbea40098a0 tx=0x7fbea4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e8034476bd0141b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:03.795 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.833+0000 7fbeafe00700 1 -- 172.21.15.12:0/4100376042 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbea4004160 con 0x7fbeb007cd70 2024-03-17T23:47:03.795 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.833+0000 7fbeafe00700 1 -- 172.21.15.12:0/4100376042 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbea4004300 con 0x7fbeb007cd70 2024-03-17T23:47:03.795 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.833+0000 7fbeafe00700 1 -- 172.21.15.12:0/4100376042 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbea4011b20 con 0x7fbeb007cd70 2024-03-17T23:47:03.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.833+0000 7fbeb8600700 1 -- 172.21.15.12:0/4100376042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 msgr2=0x7fbeb007b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.833+0000 7fbeb8600700 1 --2- 172.21.15.12:0/4100376042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 0x7fbeb007b1d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbea40098a0 tx=0x7fbea4009bb0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.833+0000 7fbeb8600700 1 -- 172.21.15.12:0/4100376042 shutdown_connections 2024-03-17T23:47:03.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.833+0000 7fbeb8600700 1 --2- 172.21.15.12:0/4100376042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 0x7fbeb007b1d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.833+0000 7fbeb8600700 1 -- 172.21.15.12:0/4100376042 >> 172.21.15.12:0/4100376042 conn(0x7fbeb01031f0 msgr2=0x7fbeb0105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:03.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.834+0000 7fbeb8600700 1 -- 172.21.15.12:0/4100376042 shutdown_connections 2024-03-17T23:47:03.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.834+0000 7fbeb8600700 1 -- 172.21.15.12:0/4100376042 wait complete. 2024-03-17T23:47:03.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.835+0000 7fbeb8600700 1 Processor -- start 2024-03-17T23:47:03.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.836+0000 7fbeb8600700 1 -- start start 2024-03-17T23:47:03.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.836+0000 7fbeb8600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 0x7fbeb019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.796 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.836+0000 7fbeb8600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb010e2d0 con 0x7fbeb007cd70 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.836+0000 7fbeb5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 0x7fbeb019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.836+0000 7fbeb5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 0x7fbeb019dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44550/0 (socket says 172.21.15.12:44550) 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.836+0000 7fbeb5c00700 1 -- 172.21.15.12:0/390520505 learned_addr learned my addr 172.21.15.12:0/390520505 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.837+0000 7fbeb5c00700 1 -- 172.21.15.12:0/390520505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb019e2e0 con 0x7fbeb007cd70 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.837+0000 7fbeb5c00700 1 --2- 172.21.15.12:0/390520505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 0x7fbeb019dda0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbea40117e0 tx=0x7fbea4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.837+0000 7fbeae000700 1 -- 172.21.15.12:0/390520505 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbea4003ba0 con 0x7fbeb007cd70 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.837+0000 7fbeae000700 1 -- 172.21.15.12:0/390520505 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbea401a910 con 0x7fbeb007cd70 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.837+0000 7fbeae000700 1 -- 172.21.15.12:0/390520505 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbea40248f0 con 0x7fbeb007cd70 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.837+0000 7fbeb8600700 1 -- 172.21.15.12:0/390520505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb019e570 con 0x7fbeb007cd70 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.838+0000 7fbeb8600700 1 -- 172.21.15.12:0/390520505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb019e9d0 con 0x7fbeb007cd70 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.839+0000 7fbeae000700 1 -- 172.21.15.12:0/390520505 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fbea4022020 con 0x7fbeb007cd70 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.839+0000 7fbeb8600700 1 -- 172.21.15.12:0/390520505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb0063900 con 0x7fbeb007cd70 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.839+0000 7fbeae000700 1 --2- 172.21.15.12:0/390520505 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe98039010 0x7fbe9803b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.797 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.839+0000 7fbeae000700 1 -- 172.21.15.12:0/390520505 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbea404d9a0 con 0x7fbeb007cd70 2024-03-17T23:47:03.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.840+0000 7fbeb5200700 1 --2- 172.21.15.12:0/390520505 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe98039010 0x7fbe9803b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.840+0000 7fbeb5200700 1 --2- 172.21.15.12:0/390520505 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe98039010 0x7fbe9803b4d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c006fd0 tx=0x7fbe9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:03.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:02.845+0000 7fbeae000700 1 -- 172.21.15.12:0/390520505 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbea401ab70 con 0x7fbeb007cd70 2024-03-17T23:47:03.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:03.084+0000 7fbeb8600700 1 -- 172.21.15.12:0/390520505 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fbeb01a13a0 con 0x7fbeb007cd70 2024-03-17T23:47:03.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:03.085+0000 7fbeae000700 1 -- 172.21.15.12:0/390520505 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fbea4010380 con 0x7fbeb007cd70 2024-03-17T23:47:03.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:03.087+0000 7fbeb8600700 1 -- 172.21.15.12:0/390520505 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe98039010 msgr2=0x7fbe9803b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:03.087+0000 7fbeb8600700 1 --2- 172.21.15.12:0/390520505 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe98039010 0x7fbe9803b4d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c006fd0 tx=0x7fbe9c006e40 comp rx=0 tx=0).stop 2024-03-17T23:47:03.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:03.087+0000 7fbeb8600700 1 -- 172.21.15.12:0/390520505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 msgr2=0x7fbeb019dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:03.087+0000 7fbeb8600700 1 --2- 172.21.15.12:0/390520505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 0x7fbeb019dda0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbea40117e0 tx=0x7fbea4011ba0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:03.088+0000 7fbeb8600700 1 -- 172.21.15.12:0/390520505 shutdown_connections 2024-03-17T23:47:03.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:03.088+0000 7fbeb8600700 1 --2- 172.21.15.12:0/390520505 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe98039010 0x7fbe9803b4d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:03.088+0000 7fbeb8600700 1 --2- 172.21.15.12:0/390520505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb007cd70 0x7fbeb019dda0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:03.088+0000 7fbeb8600700 1 -- 172.21.15.12:0/390520505 >> 172.21.15.12:0/390520505 conn(0x7fbeb01031f0 msgr2=0x7fbeb0103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:03.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:03.088+0000 7fbeb8600700 1 -- 172.21.15.12:0/390520505 shutdown_connections 2024-03-17T23:47:03.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:03.088+0000 7fbeb8600700 1 -- 172.21.15.12:0/390520505 wait complete. 2024-03-17T23:47:05.351 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.394+0000 7f4498000700 1 Processor -- start 2024-03-17T23:47:05.351 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.394+0000 7f4498000700 1 -- start start 2024-03-17T23:47:05.352 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.395+0000 7f4498000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 0x7f449010a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.352 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.395+0000 7f4498000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f449010a7d0 con 0x7f4490109e70 2024-03-17T23:47:05.352 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.395+0000 7f4495600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 0x7f449010a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.352 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.395+0000 7f4495600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 0x7f449010a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46742/0 (socket says 172.21.15.12:46742) 2024-03-17T23:47:05.352 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.395+0000 7f4495600700 1 -- 172.21.15.12:0/1635059422 learned_addr learned my addr 172.21.15.12:0/1635059422 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:47:05.352 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.396+0000 7f4495600700 1 -- 172.21.15.12:0/1635059422 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f449010afd0 con 0x7f4490109e70 2024-03-17T23:47:05.352 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.396+0000 7f4495600700 1 --2- 172.21.15.12:0/1635059422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 0x7f449010a290 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f44840098a0 tx=0x7f4484009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3056950e477572d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:05.352 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.396+0000 7f448fe00700 1 -- 172.21.15.12:0/1635059422 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4484004160 con 0x7f4490109e70 2024-03-17T23:47:05.352 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.396+0000 7f448fe00700 1 -- 172.21.15.12:0/1635059422 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4484004300 con 0x7f4490109e70 2024-03-17T23:47:05.352 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.396+0000 7f448fe00700 1 -- 172.21.15.12:0/1635059422 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4484011b20 con 0x7f4490109e70 2024-03-17T23:47:05.352 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.397+0000 7f4498000700 1 -- 172.21.15.12:0/1635059422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 msgr2=0x7f449010a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.352 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.397+0000 7f4498000700 1 --2- 172.21.15.12:0/1635059422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 0x7f449010a290 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f44840098a0 tx=0x7f4484009bb0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.353 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.397+0000 7f4498000700 1 -- 172.21.15.12:0/1635059422 shutdown_connections 2024-03-17T23:47:05.353 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.397+0000 7f4498000700 1 --2- 172.21.15.12:0/1635059422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 0x7f449010a290 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.353 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.397+0000 7f4498000700 1 -- 172.21.15.12:0/1635059422 >> 172.21.15.12:0/1635059422 conn(0x7f449007d230 msgr2=0x7f449007d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:05.357 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.397+0000 7f4498000700 1 -- 172.21.15.12:0/1635059422 shutdown_connections 2024-03-17T23:47:05.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.397+0000 7f4498000700 1 -- 172.21.15.12:0/1635059422 wait complete. 2024-03-17T23:47:05.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.398+0000 7f4498000700 1 Processor -- start 2024-03-17T23:47:05.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.399+0000 7f4498000700 1 -- start start 2024-03-17T23:47:05.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.399+0000 7f4498000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 0x7f449019e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.399+0000 7f4498000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f449010e410 con 0x7f4490109e70 2024-03-17T23:47:05.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.399+0000 7f4495600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 0x7f449019e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.399+0000 7f4495600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 0x7f449019e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46744/0 (socket says 172.21.15.12:46744) 2024-03-17T23:47:05.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.399+0000 7f4495600700 1 -- 172.21.15.12:0/1428885089 learned_addr learned my addr 172.21.15.12:0/1428885089 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:47:05.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.400+0000 7f4495600700 1 -- 172.21.15.12:0/1428885089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f449019e5d0 con 0x7f4490109e70 2024-03-17T23:47:05.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.400+0000 7f4495600700 1 --2- 172.21.15.12:0/1428885089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 0x7f449019e090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f44840117e0 tx=0x7f4484011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.400+0000 7f448e000700 1 -- 172.21.15.12:0/1428885089 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4484003ba0 con 0x7f4490109e70 2024-03-17T23:47:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.400+0000 7f448e000700 1 -- 172.21.15.12:0/1428885089 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f448401a910 con 0x7f4490109e70 2024-03-17T23:47:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.400+0000 7f4498000700 1 -- 172.21.15.12:0/1428885089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f449019e860 con 0x7f4490109e70 2024-03-17T23:47:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.400+0000 7f448e000700 1 -- 172.21.15.12:0/1428885089 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44840248f0 con 0x7f4490109e70 2024-03-17T23:47:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.401+0000 7f4498000700 1 -- 172.21.15.12:0/1428885089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f449019ecc0 con 0x7f4490109e70 2024-03-17T23:47:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.402+0000 7f448e000700 1 -- 172.21.15.12:0/1428885089 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f4484022020 con 0x7f4490109e70 2024-03-17T23:47:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.402+0000 7f4498000700 1 -- 172.21.15.12:0/1428885089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4490063900 con 0x7f4490109e70 2024-03-17T23:47:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.402+0000 7f448e000700 1 --2- 172.21.15.12:0/1428885089 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f4478039070 0x7f447803b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.402+0000 7f448e000700 1 -- 172.21.15.12:0/1428885089 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f448404d460 con 0x7f4490109e70 2024-03-17T23:47:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.403+0000 7f4494c00700 1 --2- 172.21.15.12:0/1428885089 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f4478039070 0x7f447803b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.404+0000 7f4494c00700 1 --2- 172.21.15.12:0/1428885089 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f4478039070 0x7f447803b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f447c006fd0 tx=0x7f447c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.408+0000 7f448e000700 1 -- 172.21.15.12:0/1428885089 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f448401ab70 con 0x7f4490109e70 2024-03-17T23:47:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.739+0000 7f4498000700 1 -- 172.21.15.12:0/1428885089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f44901a17f0 con 0x7f4490109e70 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.743+0000 7f448e000700 1 -- 172.21.15.12:0/1428885089 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f4484010380 con 0x7f4490109e70 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.745+0000 7f4498000700 1 -- 172.21.15.12:0/1428885089 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f4478039070 msgr2=0x7f447803b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.745+0000 7f4498000700 1 --2- 172.21.15.12:0/1428885089 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f4478039070 0x7f447803b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f447c006fd0 tx=0x7f447c006e40 comp rx=0 tx=0).stop 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.745+0000 7f4498000700 1 -- 172.21.15.12:0/1428885089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 msgr2=0x7f449019e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.745+0000 7f4498000700 1 --2- 172.21.15.12:0/1428885089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 0x7f449019e090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f44840117e0 tx=0x7f4484011ba0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.746+0000 7f4498000700 1 -- 172.21.15.12:0/1428885089 shutdown_connections 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.746+0000 7f4498000700 1 --2- 172.21.15.12:0/1428885089 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f4478039070 0x7f447803b530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.746+0000 7f4498000700 1 --2- 172.21.15.12:0/1428885089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4490109e70 0x7f449019e090 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.746+0000 7f4498000700 1 -- 172.21.15.12:0/1428885089 >> 172.21.15.12:0/1428885089 conn(0x7f449007d230 msgr2=0x7f44901073d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.746+0000 7f4498000700 1 -- 172.21.15.12:0/1428885089 shutdown_connections 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-17T23:47:04.746+0000 7f4498000700 1 -- 172.21.15.12:0/1428885089 wait complete. 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-17T23:47:05.360 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout:Or, if you are only running a single cluster on this host: 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout: ceph telemetry on 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout:For more information see: 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:47:05.361 INFO:teuthology.orchestra.run.smithi012.stdout:Bootstrap complete. 2024-03-17T23:47:05.453 INFO:tasks.cephadm:Fetching config... 2024-03-17T23:47:05.454 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:47:05.454 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-17T23:47:05.499 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-17T23:47:05.499 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:47:05.499 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-17T23:47:05.582 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-17T23:47:05.582 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:47:05.583 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-17T23:47:05.716 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-17T23:47:05.716 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:47:05.716 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-17T23:47:05.769 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-17T23:47:05.769 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOxBkCPX3tVGgoNag9XTh7NfPmwRKi8uPR3kx95S5gsS8mq26AcRbk2GjbnofOpger2V8+1QBrrf3ATtwO9K6VdFHH62B4at28pDel5r1smBZIasWp6f2KR9z/j1QPJbFGlQPBgFZ4nlvLxWUe9KjKxMngCIGTOJa23BQ5JR0+sJchid2r003pfpSmDsnKKRUs7RdDskCuF84owTDqqUSGFRsPzDMytmlJwqO50AstRf/lfMy6filOFo4E224dJ6ld4Qw5LhfCpumeAVvrtVCezp/CkV9oAiH6T66vwO6rtdr2UjPW2NtNJL28IAQw1WTbiwx1jKwyKpdsyGvttEX2j6Anj6g7A6Tr0MULPui7gHBynv5NcZsY9xvfS+DE6+e2eNpPAWDpCjUQuKZikxnv/1kF1bOaSThnSRXjQP9UBgn2nMZJbTO9C0RA/+tnGmU/KGVcxTY7/kiTbrd4wtyME/9317QbZtCns9VmeSNtbVCcKrOh2nbTU10YrmWywis= ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-17T23:47:05.886 INFO:teuthology.orchestra.run.smithi012.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOxBkCPX3tVGgoNag9XTh7NfPmwRKi8uPR3kx95S5gsS8mq26AcRbk2GjbnofOpger2V8+1QBrrf3ATtwO9K6VdFHH62B4at28pDel5r1smBZIasWp6f2KR9z/j1QPJbFGlQPBgFZ4nlvLxWUe9KjKxMngCIGTOJa23BQ5JR0+sJchid2r003pfpSmDsnKKRUs7RdDskCuF84owTDqqUSGFRsPzDMytmlJwqO50AstRf/lfMy6filOFo4E224dJ6ld4Qw5LhfCpumeAVvrtVCezp/CkV9oAiH6T66vwO6rtdr2UjPW2NtNJL28IAQw1WTbiwx1jKwyKpdsyGvttEX2j6Anj6g7A6Tr0MULPui7gHBynv5NcZsY9xvfS+DE6+e2eNpPAWDpCjUQuKZikxnv/1kF1bOaSThnSRXjQP9UBgn2nMZJbTO9C0RA/+tnGmU/KGVcxTY7/kiTbrd4wtyME/9317QbZtCns9VmeSNtbVCcKrOh2nbTU10YrmWywis= ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:47:05.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:05 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1428885089' entity='client.admin' 2024-03-17T23:47:05.951 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOxBkCPX3tVGgoNag9XTh7NfPmwRKi8uPR3kx95S5gsS8mq26AcRbk2GjbnofOpger2V8+1QBrrf3ATtwO9K6VdFHH62B4at28pDel5r1smBZIasWp6f2KR9z/j1QPJbFGlQPBgFZ4nlvLxWUe9KjKxMngCIGTOJa23BQ5JR0+sJchid2r003pfpSmDsnKKRUs7RdDskCuF84owTDqqUSGFRsPzDMytmlJwqO50AstRf/lfMy6filOFo4E224dJ6ld4Qw5LhfCpumeAVvrtVCezp/CkV9oAiH6T66vwO6rtdr2UjPW2NtNJL28IAQw1WTbiwx1jKwyKpdsyGvttEX2j6Anj6g7A6Tr0MULPui7gHBynv5NcZsY9xvfS+DE6+e2eNpPAWDpCjUQuKZikxnv/1kF1bOaSThnSRXjQP9UBgn2nMZJbTO9C0RA/+tnGmU/KGVcxTY7/kiTbrd4wtyME/9317QbZtCns9VmeSNtbVCcKrOh2nbTU10YrmWywis= ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-17T23:47:05.997 INFO:teuthology.orchestra.run.smithi079.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOxBkCPX3tVGgoNag9XTh7NfPmwRKi8uPR3kx95S5gsS8mq26AcRbk2GjbnofOpger2V8+1QBrrf3ATtwO9K6VdFHH62B4at28pDel5r1smBZIasWp6f2KR9z/j1QPJbFGlQPBgFZ4nlvLxWUe9KjKxMngCIGTOJa23BQ5JR0+sJchid2r003pfpSmDsnKKRUs7RdDskCuF84owTDqqUSGFRsPzDMytmlJwqO50AstRf/lfMy6filOFo4E224dJ6ld4Qw5LhfCpumeAVvrtVCezp/CkV9oAiH6T66vwO6rtdr2UjPW2NtNJL28IAQw1WTbiwx1jKwyKpdsyGvttEX2j6Anj6g7A6Tr0MULPui7gHBynv5NcZsY9xvfS+DE6+e2eNpPAWDpCjUQuKZikxnv/1kF1bOaSThnSRXjQP9UBgn2nMZJbTO9C0RA/+tnGmU/KGVcxTY7/kiTbrd4wtyME/9317QbZtCns9VmeSNtbVCcKrOh2nbTU10YrmWywis= ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:47:06.012 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOxBkCPX3tVGgoNag9XTh7NfPmwRKi8uPR3kx95S5gsS8mq26AcRbk2GjbnofOpger2V8+1QBrrf3ATtwO9K6VdFHH62B4at28pDel5r1smBZIasWp6f2KR9z/j1QPJbFGlQPBgFZ4nlvLxWUe9KjKxMngCIGTOJa23BQ5JR0+sJchid2r003pfpSmDsnKKRUs7RdDskCuF84owTDqqUSGFRsPzDMytmlJwqO50AstRf/lfMy6filOFo4E224dJ6ld4Qw5LhfCpumeAVvrtVCezp/CkV9oAiH6T66vwO6rtdr2UjPW2NtNJL28IAQw1WTbiwx1jKwyKpdsyGvttEX2j6Anj6g7A6Tr0MULPui7gHBynv5NcZsY9xvfS+DE6+e2eNpPAWDpCjUQuKZikxnv/1kF1bOaSThnSRXjQP9UBgn2nMZJbTO9C0RA/+tnGmU/KGVcxTY7/kiTbrd4wtyME/9317QbZtCns9VmeSNtbVCcKrOh2nbTU10YrmWywis= ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-17T23:47:06.057 INFO:teuthology.orchestra.run.smithi191.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOxBkCPX3tVGgoNag9XTh7NfPmwRKi8uPR3kx95S5gsS8mq26AcRbk2GjbnofOpger2V8+1QBrrf3ATtwO9K6VdFHH62B4at28pDel5r1smBZIasWp6f2KR9z/j1QPJbFGlQPBgFZ4nlvLxWUe9KjKxMngCIGTOJa23BQ5JR0+sJchid2r003pfpSmDsnKKRUs7RdDskCuF84owTDqqUSGFRsPzDMytmlJwqO50AstRf/lfMy6filOFo4E224dJ6ld4Qw5LhfCpumeAVvrtVCezp/CkV9oAiH6T66vwO6rtdr2UjPW2NtNJL28IAQw1WTbiwx1jKwyKpdsyGvttEX2j6Anj6g7A6Tr0MULPui7gHBynv5NcZsY9xvfS+DE6+e2eNpPAWDpCjUQuKZikxnv/1kF1bOaSThnSRXjQP9UBgn2nMZJbTO9C0RA/+tnGmU/KGVcxTY7/kiTbrd4wtyME/9317QbZtCns9VmeSNtbVCcKrOh2nbTU10YrmWywis= ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:47:06.072 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-17T23:47:06.501 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:47:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.616+0000 7f4b7f600700 1 -- 172.21.15.12:0/1579120692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b78102790 msgr2=0x7f4b78104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.616+0000 7f4b7f600700 1 --2- 172.21.15.12:0/1579120692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b78102790 0x7f4b78104b80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009910 tx=0x7f4b68009c20 comp rx=0 tx=0).stop 2024-03-17T23:47:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.616+0000 7f4b7f600700 1 -- 172.21.15.12:0/1579120692 shutdown_connections 2024-03-17T23:47:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.616+0000 7f4b7f600700 1 --2- 172.21.15.12:0/1579120692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b78102790 0x7f4b78104b80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.616+0000 7f4b7f600700 1 -- 172.21.15.12:0/1579120692 >> 172.21.15.12:0/1579120692 conn(0x7f4b780fc1a0 msgr2=0x7f4b780fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.616+0000 7f4b7f600700 1 -- 172.21.15.12:0/1579120692 shutdown_connections 2024-03-17T23:47:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.616+0000 7f4b7f600700 1 -- 172.21.15.12:0/1579120692 wait complete. 2024-03-17T23:47:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.618+0000 7f4b7f600700 1 Processor -- start 2024-03-17T23:47:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.618+0000 7f4b7f600700 1 -- start start 2024-03-17T23:47:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.619+0000 7f4b7f600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b78102790 0x7f4b78197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.619+0000 7f4b7f600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4b78107d30 con 0x7f4b78102790 2024-03-17T23:47:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.619+0000 7f4b7cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b78102790 0x7f4b78197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.619+0000 7f4b7cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b78102790 0x7f4b78197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46758/0 (socket says 172.21.15.12:46758) 2024-03-17T23:47:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.619+0000 7f4b7cc00700 1 -- 172.21.15.12:0/3666722271 learned_addr learned my addr 172.21.15.12:0/3666722271 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:47:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.620+0000 7f4b7cc00700 1 -- 172.21.15.12:0/3666722271 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b781978b0 con 0x7f4b78102790 2024-03-17T23:47:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.620+0000 7f4b7cc00700 1 --2- 172.21.15.12:0/3666722271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b78102790 0x7f4b78197370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4b68005950 tx=0x7f4b68005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.620+0000 7f4b76000700 1 -- 172.21.15.12:0/3666722271 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b68026070 con 0x7f4b78102790 2024-03-17T23:47:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.620+0000 7f4b76000700 1 -- 172.21.15.12:0/3666722271 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b6800bbd0 con 0x7f4b78102790 2024-03-17T23:47:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.620+0000 7f4b7f600700 1 -- 172.21.15.12:0/3666722271 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b78197ba0 con 0x7f4b78102790 2024-03-17T23:47:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.620+0000 7f4b76000700 1 -- 172.21.15.12:0/3666722271 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b68003a80 con 0x7f4b78102790 2024-03-17T23:47:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.620+0000 7f4b7f600700 1 -- 172.21.15.12:0/3666722271 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b7819a830 con 0x7f4b78102790 2024-03-17T23:47:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.622+0000 7f4b76000700 1 -- 172.21.15.12:0/3666722271 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f4b68029070 con 0x7f4b78102790 2024-03-17T23:47:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.622+0000 7f4b7f600700 1 -- 172.21.15.12:0/3666722271 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b781084b0 con 0x7f4b78102790 2024-03-17T23:47:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.622+0000 7f4b76000700 1 --2- 172.21.15.12:0/3666722271 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f4b5c038fc0 0x7f4b5c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.622+0000 7f4b76000700 1 -- 172.21.15.12:0/3666722271 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4b6801e070 con 0x7f4b78102790 2024-03-17T23:47:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.622+0000 7f4b6fe00700 1 --2- 172.21.15.12:0/3666722271 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f4b5c038fc0 0x7f4b5c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.623+0000 7f4b6fe00700 1 --2- 172.21.15.12:0/3666722271 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f4b5c038fc0 0x7f4b5c03b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4b60006fd0 tx=0x7f4b60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.628+0000 7f4b76000700 1 -- 172.21.15.12:0/3666722271 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4b68013070 con 0x7f4b78102790 2024-03-17T23:47:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.858+0000 7f4b7f600700 1 -- 172.21.15.12:0/3666722271 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f4b7802dd60 con 0x7f4b78102790 2024-03-17T23:47:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.865+0000 7f4b76000700 1 -- 172.21.15.12:0/3666722271 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f4b68030020 con 0x7f4b78102790 2024-03-17T23:47:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.868+0000 7f4b7f600700 1 -- 172.21.15.12:0/3666722271 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f4b5c038fc0 msgr2=0x7f4b5c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.868+0000 7f4b7f600700 1 --2- 172.21.15.12:0/3666722271 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f4b5c038fc0 0x7f4b5c03b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4b60006fd0 tx=0x7f4b60006e40 comp rx=0 tx=0).stop 2024-03-17T23:47:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.868+0000 7f4b7f600700 1 -- 172.21.15.12:0/3666722271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b78102790 msgr2=0x7f4b78197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.868+0000 7f4b7f600700 1 --2- 172.21.15.12:0/3666722271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b78102790 0x7f4b78197370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4b68005950 tx=0x7f4b68005030 comp rx=0 tx=0).stop 2024-03-17T23:47:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.868+0000 7f4b7f600700 1 -- 172.21.15.12:0/3666722271 shutdown_connections 2024-03-17T23:47:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.868+0000 7f4b7f600700 1 --2- 172.21.15.12:0/3666722271 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f4b5c038fc0 0x7f4b5c03b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.868+0000 7f4b7f600700 1 --2- 172.21.15.12:0/3666722271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b78102790 0x7f4b78197370 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.868+0000 7f4b7f600700 1 -- 172.21.15.12:0/3666722271 >> 172.21.15.12:0/3666722271 conn(0x7f4b780fc1a0 msgr2=0x7f4b780fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.869+0000 7f4b7f600700 1 -- 172.21.15.12:0/3666722271 shutdown_connections 2024-03-17T23:47:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:08.869+0000 7f4b7f600700 1 -- 172.21.15.12:0/3666722271 wait complete. 2024-03-17T23:47:09.624 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-17T23:47:09.624 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-17T23:47:10.063 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:47:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi012", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:47:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:09 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3666722271' entity='client.admin' 2024-03-17T23:47:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:47:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:12.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.194+0000 7ff5a6200700 1 -- 172.21.15.12:0/3800726589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5a0076300 msgr2=0x7ff5a0074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.194+0000 7ff5a6200700 1 --2- 172.21.15.12:0/3800726589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5a0076300 0x7ff5a0074760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff590009910 tx=0x7ff590009c20 comp rx=0 tx=0).stop 2024-03-17T23:47:12.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.194+0000 7ff5a6200700 1 -- 172.21.15.12:0/3800726589 shutdown_connections 2024-03-17T23:47:12.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.194+0000 7ff5a6200700 1 --2- 172.21.15.12:0/3800726589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5a0076300 0x7ff5a0074760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.194+0000 7ff5a6200700 1 -- 172.21.15.12:0/3800726589 >> 172.21.15.12:0/3800726589 conn(0x7ff5a00fd2a0 msgr2=0x7ff5a00ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:12.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.194+0000 7ff5a6200700 1 -- 172.21.15.12:0/3800726589 shutdown_connections 2024-03-17T23:47:12.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.194+0000 7ff5a6200700 1 -- 172.21.15.12:0/3800726589 wait complete. 2024-03-17T23:47:12.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.196+0000 7ff5a6200700 1 Processor -- start 2024-03-17T23:47:12.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.196+0000 7ff5a6200700 1 -- start start 2024-03-17T23:47:12.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.196+0000 7ff5a6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5a0076300 0x7ff5a019da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.196+0000 7ff5a6200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a010e460 con 0x7ff5a0076300 2024-03-17T23:47:12.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.196+0000 7ff59f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5a0076300 0x7ff5a019da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.196+0000 7ff59f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5a0076300 0x7ff5a019da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46772/0 (socket says 172.21.15.12:46772) 2024-03-17T23:47:12.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.196+0000 7ff59f400700 1 -- 172.21.15.12:0/3317682047 learned_addr learned my addr 172.21.15.12:0/3317682047 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:47:12.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.197+0000 7ff59f400700 1 -- 172.21.15.12:0/3317682047 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a019df40 con 0x7ff5a0076300 2024-03-17T23:47:12.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.197+0000 7ff59f400700 1 --2- 172.21.15.12:0/3317682047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5a0076300 0x7ff5a019da00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff590005850 tx=0x7ff590005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:12.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.197+0000 7ff597e00700 1 -- 172.21.15.12:0/3317682047 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff590026070 con 0x7ff5a0076300 2024-03-17T23:47:12.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.197+0000 7ff597e00700 1 -- 172.21.15.12:0/3317682047 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5900044a0 con 0x7ff5a0076300 2024-03-17T23:47:12.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.198+0000 7ff5a6200700 1 -- 172.21.15.12:0/3317682047 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a019e1d0 con 0x7ff5a0076300 2024-03-17T23:47:12.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.198+0000 7ff597e00700 1 -- 172.21.15.12:0/3317682047 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff590003ac0 con 0x7ff5a0076300 2024-03-17T23:47:12.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.198+0000 7ff5a6200700 1 -- 172.21.15.12:0/3317682047 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a01a0ec0 con 0x7ff5a0076300 2024-03-17T23:47:12.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.199+0000 7ff5a6200700 1 -- 172.21.15.12:0/3317682047 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5a010eb10 con 0x7ff5a0076300 2024-03-17T23:47:12.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.199+0000 7ff597e00700 1 -- 172.21.15.12:0/3317682047 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ff590029070 con 0x7ff5a0076300 2024-03-17T23:47:12.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.200+0000 7ff597e00700 1 --2- 172.21.15.12:0/3317682047 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff584038fc0 0x7ff58403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.200+0000 7ff597e00700 1 -- 172.21.15.12:0/3317682047 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff59001e070 con 0x7ff5a0076300 2024-03-17T23:47:12.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.200+0000 7ff59ea00700 1 --2- 172.21.15.12:0/3317682047 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff584038fc0 0x7ff58403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.201+0000 7ff59ea00700 1 --2- 172.21.15.12:0/3317682047 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff584038fc0 0x7ff58403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff58c006fd0 tx=0x7ff58c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:12.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.205+0000 7ff597e00700 1 -- 172.21.15.12:0/3317682047 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff590013070 con 0x7ff5a0076300 2024-03-17T23:47:12.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.438+0000 7ff5a6200700 1 -- 172.21.15.12:0/3317682047 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7ff5a002dc80 con 0x7ff584038fc0 2024-03-17T23:47:12.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.444+0000 7ff597e00700 1 -- 172.21.15.12:0/3317682047 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff5a002dc80 con 0x7ff584038fc0 2024-03-17T23:47:12.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.447+0000 7ff5a6200700 1 -- 172.21.15.12:0/3317682047 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff584038fc0 msgr2=0x7ff58403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.447+0000 7ff5a6200700 1 --2- 172.21.15.12:0/3317682047 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff584038fc0 0x7ff58403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff58c006fd0 tx=0x7ff58c006e40 comp rx=0 tx=0).stop 2024-03-17T23:47:12.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.447+0000 7ff5a6200700 1 -- 172.21.15.12:0/3317682047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5a0076300 msgr2=0x7ff5a019da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.447+0000 7ff5a6200700 1 --2- 172.21.15.12:0/3317682047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5a0076300 0x7ff5a019da00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff590005850 tx=0x7ff590005030 comp rx=0 tx=0).stop 2024-03-17T23:47:12.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.447+0000 7ff5a6200700 1 -- 172.21.15.12:0/3317682047 shutdown_connections 2024-03-17T23:47:12.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.447+0000 7ff5a6200700 1 --2- 172.21.15.12:0/3317682047 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff584038fc0 0x7ff58403b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.447+0000 7ff5a6200700 1 --2- 172.21.15.12:0/3317682047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5a0076300 0x7ff5a019da00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.447+0000 7ff5a6200700 1 -- 172.21.15.12:0/3317682047 >> 172.21.15.12:0/3317682047 conn(0x7ff5a00fd2a0 msgr2=0x7ff5a00fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:12.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.448+0000 7ff5a6200700 1 -- 172.21.15.12:0/3317682047 shutdown_connections 2024-03-17T23:47:12.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:12.448+0000 7ff5a6200700 1 -- 172.21.15.12:0/3317682047 wait complete. 2024-03-17T23:47:13.289 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi079 2024-03-17T23:47:13.290 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-17T23:47:13.290 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/etc/ceph/ceph.conf 2024-03-17T23:47:13.307 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-17T23:47:13.308 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-17T23:47:13.364 INFO:tasks.cephadm:Adding host smithi079 to orchestrator... 2024-03-17T23:47:13.365 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch host add smithi079 2024-03-17T23:47:13.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:13 smithi012 ceph-mon[31541]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:47:13.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:13.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:47:13.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:47:13.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:47:13.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:13 smithi012 ceph-mon[31541]: Updating smithi012:/etc/ceph/ceph.conf 2024-03-17T23:47:13.795 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:47:14.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:14 smithi012 ceph-mon[31541]: Updating smithi012:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:47:14.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:14 smithi012 ceph-mon[31541]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-03-17T23:47:15.527 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:15 smithi012 ceph-mon[31541]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:15.527 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:15 smithi012 ceph-mon[31541]: Updating smithi012:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-17T23:47:15.527 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:15.527 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:15.527 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:16.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.015+0000 7f3a73600700 1 -- 172.21.15.12:0/2263026772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a6c104130 msgr2=0x7f3a6c104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.015+0000 7f3a73600700 1 --2- 172.21.15.12:0/2263026772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a6c104130 0x7f3a6c104510 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009910 tx=0x7f3a60009c20 comp rx=0 tx=0).stop 2024-03-17T23:47:16.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.015+0000 7f3a73600700 1 -- 172.21.15.12:0/2263026772 shutdown_connections 2024-03-17T23:47:16.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.015+0000 7f3a73600700 1 --2- 172.21.15.12:0/2263026772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a6c104130 0x7f3a6c104510 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.015+0000 7f3a73600700 1 -- 172.21.15.12:0/2263026772 >> 172.21.15.12:0/2263026772 conn(0x7f3a6c0ff9c0 msgr2=0x7f3a6c101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.015+0000 7f3a73600700 1 -- 172.21.15.12:0/2263026772 shutdown_connections 2024-03-17T23:47:16.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.015+0000 7f3a73600700 1 -- 172.21.15.12:0/2263026772 wait complete. 2024-03-17T23:47:16.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.016+0000 7f3a73600700 1 Processor -- start 2024-03-17T23:47:16.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.017+0000 7f3a73600700 1 -- start start 2024-03-17T23:47:16.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.017+0000 7f3a73600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a6c104130 0x7f3a6c19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.017+0000 7f3a73600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6c10e500 con 0x7f3a6c104130 2024-03-17T23:47:16.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.017+0000 7f3a70c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a6c104130 0x7f3a6c19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.017+0000 7f3a70c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a6c104130 0x7f3a6c19f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52144/0 (socket says 172.21.15.12:52144) 2024-03-17T23:47:16.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.017+0000 7f3a70c00700 1 -- 172.21.15.12:0/4136547501 learned_addr learned my addr 172.21.15.12:0/4136547501 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:47:16.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.018+0000 7f3a70c00700 1 -- 172.21.15.12:0/4136547501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a6c19fbb0 con 0x7f3a6c104130 2024-03-17T23:47:16.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.018+0000 7f3a70c00700 1 --2- 172.21.15.12:0/4136547501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a6c104130 0x7f3a6c19f670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3a60006010 tx=0x7f3a60005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.018+0000 7f3a69600700 1 -- 172.21.15.12:0/4136547501 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a60026070 con 0x7f3a6c104130 2024-03-17T23:47:16.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.018+0000 7f3a73600700 1 -- 172.21.15.12:0/4136547501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a6c19fea0 con 0x7f3a6c104130 2024-03-17T23:47:16.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.018+0000 7f3a69600700 1 -- 172.21.15.12:0/4136547501 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6000bbd0 con 0x7f3a6c104130 2024-03-17T23:47:16.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.018+0000 7f3a69600700 1 -- 172.21.15.12:0/4136547501 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a60003910 con 0x7f3a6c104130 2024-03-17T23:47:16.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.019+0000 7f3a73600700 1 -- 172.21.15.12:0/4136547501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a6c1997f0 con 0x7f3a6c104130 2024-03-17T23:47:16.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.020+0000 7f3a73600700 1 -- 172.21.15.12:0/4136547501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a6c105610 con 0x7f3a6c104130 2024-03-17T23:47:16.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.020+0000 7f3a69600700 1 -- 172.21.15.12:0/4136547501 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f3a60029070 con 0x7f3a6c104130 2024-03-17T23:47:16.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.020+0000 7f3a69600700 1 --2- 172.21.15.12:0/4136547501 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3a54038fc0 0x7f3a5403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.020+0000 7f3a69600700 1 -- 172.21.15.12:0/4136547501 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3a6001e070 con 0x7f3a6c104130 2024-03-17T23:47:16.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.021+0000 7f3a6be00700 1 --2- 172.21.15.12:0/4136547501 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3a54038fc0 0x7f3a5403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.021+0000 7f3a6be00700 1 --2- 172.21.15.12:0/4136547501 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3a54038fc0 0x7f3a5403b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a58006fd0 tx=0x7f3a58006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.026+0000 7f3a69600700 1 -- 172.21.15.12:0/4136547501 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3a60013070 con 0x7f3a6c104130 2024-03-17T23:47:16.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:16.256+0000 7f3a73600700 1 -- 172.21.15.12:0/4136547501 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi079", "target": ["mon-mgr", ""]}) -- 0x7f3a6c19ad20 con 0x7f3a54038fc0 2024-03-17T23:47:17.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:17 smithi012 ceph-mon[31541]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:17.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:17 smithi012 ceph-mon[31541]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi079", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:47:18.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:18 smithi012 ceph-mon[31541]: Deploying cephadm binary to smithi079 2024-03-17T23:47:19.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:19 smithi012 ceph-mon[31541]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:21.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:21.196+0000 7f3a69600700 1 -- 172.21.15.12:0/4136547501 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f3a6c19ad20 con 0x7f3a54038fc0 2024-03-17T23:47:21.199 INFO:teuthology.orchestra.run.smithi012.stdout:Added host 'smithi079' with addr '172.21.15.79' 2024-03-17T23:47:21.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:21.200+0000 7f3a73600700 1 -- 172.21.15.12:0/4136547501 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3a54038fc0 msgr2=0x7f3a5403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:21.200+0000 7f3a73600700 1 --2- 172.21.15.12:0/4136547501 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3a54038fc0 0x7f3a5403b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a58006fd0 tx=0x7f3a58006e40 comp rx=0 tx=0).stop 2024-03-17T23:47:21.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:21.200+0000 7f3a73600700 1 -- 172.21.15.12:0/4136547501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a6c104130 msgr2=0x7f3a6c19f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:21.200+0000 7f3a73600700 1 --2- 172.21.15.12:0/4136547501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a6c104130 0x7f3a6c19f670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3a60006010 tx=0x7f3a60005030 comp rx=0 tx=0).stop 2024-03-17T23:47:21.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:21.200+0000 7f3a73600700 1 -- 172.21.15.12:0/4136547501 shutdown_connections 2024-03-17T23:47:21.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:21.201+0000 7f3a73600700 1 --2- 172.21.15.12:0/4136547501 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3a54038fc0 0x7f3a5403b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:21.201+0000 7f3a73600700 1 --2- 172.21.15.12:0/4136547501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a6c104130 0x7f3a6c19f670 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:21.201+0000 7f3a73600700 1 -- 172.21.15.12:0/4136547501 >> 172.21.15.12:0/4136547501 conn(0x7f3a6c0ff9c0 msgr2=0x7f3a6c105b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:21.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:21.201+0000 7f3a73600700 1 -- 172.21.15.12:0/4136547501 shutdown_connections 2024-03-17T23:47:21.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:21.201+0000 7f3a73600700 1 -- 172.21.15.12:0/4136547501 wait complete. 2024-03-17T23:47:21.727 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:21 smithi012 ceph-mon[31541]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:21.727 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:21 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:21.727 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:21 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:47:22.131 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-17T23:47:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:47:23.029 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:22 smithi012 ceph-mon[31541]: Added host smithi079 2024-03-17T23:47:23.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:22 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:23.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:22 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:24.068 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:23 smithi012 ceph-mon[31541]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:24.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.530+0000 7faca0400700 1 -- 172.21.15.12:0/2651880978 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac98104130 msgr2=0x7fac98104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:24.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.530+0000 7faca0400700 1 --2- 172.21.15.12:0/2651880978 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac98104130 0x7fac98104510 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009910 tx=0x7fac8c009c20 comp rx=0 tx=0).stop 2024-03-17T23:47:24.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.530+0000 7faca0400700 1 -- 172.21.15.12:0/2651880978 shutdown_connections 2024-03-17T23:47:24.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.530+0000 7faca0400700 1 --2- 172.21.15.12:0/2651880978 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac98104130 0x7fac98104510 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:24.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.530+0000 7faca0400700 1 -- 172.21.15.12:0/2651880978 >> 172.21.15.12:0/2651880978 conn(0x7fac980ff9c0 msgr2=0x7fac98101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:24.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.530+0000 7faca0400700 1 -- 172.21.15.12:0/2651880978 shutdown_connections 2024-03-17T23:47:24.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.530+0000 7faca0400700 1 -- 172.21.15.12:0/2651880978 wait complete. 2024-03-17T23:47:24.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.531+0000 7faca0400700 1 Processor -- start 2024-03-17T23:47:24.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.531+0000 7faca0400700 1 -- start start 2024-03-17T23:47:24.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.531+0000 7faca0400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac98104130 0x7fac9819f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:24.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.531+0000 7faca0400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fac9810e500 con 0x7fac98104130 2024-03-17T23:47:24.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.532+0000 7fac9da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac98104130 0x7fac9819f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:24.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.532+0000 7fac9da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac98104130 0x7fac9819f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47232/0 (socket says 172.21.15.12:47232) 2024-03-17T23:47:24.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.532+0000 7fac9da00700 1 -- 172.21.15.12:0/533685347 learned_addr learned my addr 172.21.15.12:0/533685347 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:47:24.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.532+0000 7fac9da00700 1 -- 172.21.15.12:0/533685347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac9819fbb0 con 0x7fac98104130 2024-03-17T23:47:24.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.532+0000 7fac9da00700 1 --2- 172.21.15.12:0/533685347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac98104130 0x7fac9819f670 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fac8c006010 tx=0x7fac8c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:24.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.533+0000 7fac96000700 1 -- 172.21.15.12:0/533685347 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac8c026070 con 0x7fac98104130 2024-03-17T23:47:24.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.533+0000 7fac96000700 1 -- 172.21.15.12:0/533685347 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac8c00bbd0 con 0x7fac98104130 2024-03-17T23:47:24.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.533+0000 7fac96000700 1 -- 172.21.15.12:0/533685347 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac8c003910 con 0x7fac98104130 2024-03-17T23:47:24.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.533+0000 7faca0400700 1 -- 172.21.15.12:0/533685347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac9819fea0 con 0x7fac98104130 2024-03-17T23:47:24.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.533+0000 7faca0400700 1 -- 172.21.15.12:0/533685347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac981997f0 con 0x7fac98104130 2024-03-17T23:47:24.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.534+0000 7fac96000700 1 -- 172.21.15.12:0/533685347 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fac8c029070 con 0x7fac98104130 2024-03-17T23:47:24.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.534+0000 7faca0400700 1 -- 172.21.15.12:0/533685347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac98105610 con 0x7fac98104130 2024-03-17T23:47:24.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.535+0000 7fac96000700 1 --2- 172.21.15.12:0/533685347 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fac80039020 0x7fac8003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:24.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.535+0000 7fac96000700 1 -- 172.21.15.12:0/533685347 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fac8c01e070 con 0x7fac98104130 2024-03-17T23:47:24.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.535+0000 7fac9d000700 1 --2- 172.21.15.12:0/533685347 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fac80039020 0x7fac8003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:24.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.536+0000 7fac9d000700 1 --2- 172.21.15.12:0/533685347 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fac80039020 0x7fac8003b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fac84006fd0 tx=0x7fac84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.541+0000 7fac96000700 1 -- 172.21.15.12:0/533685347 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fac8c013070 con 0x7fac98104130 2024-03-17T23:47:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.773+0000 7faca0400700 1 -- 172.21.15.12:0/533685347 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fac9819ad20 con 0x7fac80039020 2024-03-17T23:47:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.776+0000 7fac96000700 1 -- 172.21.15.12:0/533685347 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7fac9819ad20 con 0x7fac80039020 2024-03-17T23:47:24.778 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:47:24.778 INFO:teuthology.orchestra.run.smithi012.stdout:[{"addr": "172.21.15.12", "hostname": "smithi012", "labels": [], "status": ""}, {"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}] 2024-03-17T23:47:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.779+0000 7faca0400700 1 -- 172.21.15.12:0/533685347 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fac80039020 msgr2=0x7fac8003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.779+0000 7faca0400700 1 --2- 172.21.15.12:0/533685347 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fac80039020 0x7fac8003b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fac84006fd0 tx=0x7fac84006e40 comp rx=0 tx=0).stop 2024-03-17T23:47:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.779+0000 7faca0400700 1 -- 172.21.15.12:0/533685347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac98104130 msgr2=0x7fac9819f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.779+0000 7faca0400700 1 --2- 172.21.15.12:0/533685347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac98104130 0x7fac9819f670 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fac8c006010 tx=0x7fac8c005030 comp rx=0 tx=0).stop 2024-03-17T23:47:24.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.779+0000 7faca0400700 1 -- 172.21.15.12:0/533685347 shutdown_connections 2024-03-17T23:47:24.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.779+0000 7faca0400700 1 --2- 172.21.15.12:0/533685347 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fac80039020 0x7fac8003b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:24.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.779+0000 7faca0400700 1 --2- 172.21.15.12:0/533685347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac98104130 0x7fac9819f670 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:24.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.779+0000 7faca0400700 1 -- 172.21.15.12:0/533685347 >> 172.21.15.12:0/533685347 conn(0x7fac980ff9c0 msgr2=0x7fac98105b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:24.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.779+0000 7faca0400700 1 -- 172.21.15.12:0/533685347 shutdown_connections 2024-03-17T23:47:24.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:24.779+0000 7faca0400700 1 -- 172.21.15.12:0/533685347 wait complete. 2024-03-17T23:47:25.130 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:24 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:25.130 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:24 smithi012 ceph-mon[31541]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:25.130 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:24 smithi012 ceph-mon[31541]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:47:25.680 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi191 2024-03-17T23:47:25.680 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-17T23:47:25.680 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.conf 2024-03-17T23:47:25.700 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-17T23:47:25.700 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-17T23:47:25.757 INFO:tasks.cephadm:Adding host smithi191 to orchestrator... 2024-03-17T23:47:25.757 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch host add smithi191 2024-03-17T23:47:26.221 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:47:27.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:27 smithi012 ceph-mon[31541]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:27.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:27.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:27.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:27.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:27.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:47:27.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:47:27.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:47:28.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.244+0000 7ffbc2600700 1 -- 172.21.15.12:0/3034267289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbbc102e60 msgr2=0x7ffbbc103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:28.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.244+0000 7ffbc2600700 1 --2- 172.21.15.12:0/3034267289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbbc102e60 0x7ffbbc103240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009910 tx=0x7ffbac009c20 comp rx=0 tx=0).stop 2024-03-17T23:47:28.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.244+0000 7ffbc2600700 1 -- 172.21.15.12:0/3034267289 shutdown_connections 2024-03-17T23:47:28.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.244+0000 7ffbc2600700 1 --2- 172.21.15.12:0/3034267289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbbc102e60 0x7ffbbc103240 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:28.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.244+0000 7ffbc2600700 1 -- 172.21.15.12:0/3034267289 >> 172.21.15.12:0/3034267289 conn(0x7ffbbc077810 msgr2=0x7ffbbc0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:28.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.244+0000 7ffbc2600700 1 -- 172.21.15.12:0/3034267289 shutdown_connections 2024-03-17T23:47:28.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.244+0000 7ffbc2600700 1 -- 172.21.15.12:0/3034267289 wait complete. 2024-03-17T23:47:28.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.245+0000 7ffbc2600700 1 Processor -- start 2024-03-17T23:47:28.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.245+0000 7ffbc2600700 1 -- start start 2024-03-17T23:47:28.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.246+0000 7ffbc2600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbbc102e60 0x7ffbbc19f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:28.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.246+0000 7ffbc2600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc10c350 con 0x7ffbbc102e60 2024-03-17T23:47:28.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.246+0000 7ffbbb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbbc102e60 0x7ffbbc19f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:28.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.246+0000 7ffbbb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbbc102e60 0x7ffbbc19f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47248/0 (socket says 172.21.15.12:47248) 2024-03-17T23:47:28.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.246+0000 7ffbbb400700 1 -- 172.21.15.12:0/422227858 learned_addr learned my addr 172.21.15.12:0/422227858 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:47:28.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.246+0000 7ffbbb400700 1 -- 172.21.15.12:0/422227858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbbc19fbe0 con 0x7ffbbc102e60 2024-03-17T23:47:28.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.247+0000 7ffbbb400700 1 --2- 172.21.15.12:0/422227858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbbc102e60 0x7ffbbc19f6a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffbac006010 tx=0x7ffbac005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:28.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.247+0000 7ffbb3e00700 1 -- 172.21.15.12:0/422227858 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbac026070 con 0x7ffbbc102e60 2024-03-17T23:47:28.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.247+0000 7ffbb3e00700 1 -- 172.21.15.12:0/422227858 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbac00bc30 con 0x7ffbbc102e60 2024-03-17T23:47:28.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.247+0000 7ffbb3e00700 1 -- 172.21.15.12:0/422227858 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbac0039a0 con 0x7ffbbc102e60 2024-03-17T23:47:28.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.247+0000 7ffbc2600700 1 -- 172.21.15.12:0/422227858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbbc19fe70 con 0x7ffbbc102e60 2024-03-17T23:47:28.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.247+0000 7ffbc2600700 1 -- 172.21.15.12:0/422227858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbbc199860 con 0x7ffbbc102e60 2024-03-17T23:47:28.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.248+0000 7ffbc2600700 1 -- 172.21.15.12:0/422227858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbbc069fa0 con 0x7ffbbc102e60 2024-03-17T23:47:28.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.249+0000 7ffbb3e00700 1 -- 172.21.15.12:0/422227858 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ffbac029070 con 0x7ffbbc102e60 2024-03-17T23:47:28.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.249+0000 7ffbb3e00700 1 --2- 172.21.15.12:0/422227858 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ffba0039020 0x7ffba003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:28.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.249+0000 7ffbb3e00700 1 -- 172.21.15.12:0/422227858 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ffbac01e070 con 0x7ffbbc102e60 2024-03-17T23:47:28.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.250+0000 7ffbbaa00700 1 --2- 172.21.15.12:0/422227858 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ffba0039020 0x7ffba003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:28.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.250+0000 7ffbbaa00700 1 --2- 172.21.15.12:0/422227858 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ffba0039020 0x7ffba003b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ffba8006fd0 tx=0x7ffba8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:28.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.255+0000 7ffbb3e00700 1 -- 172.21.15.12:0/422227858 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffbac013070 con 0x7ffbbc102e60 2024-03-17T23:47:28.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:28 smithi012 ceph-mon[31541]: Updating smithi079:/etc/ceph/ceph.conf 2024-03-17T23:47:28.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:28 smithi012 ceph-mon[31541]: Updating smithi079:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:47:28.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:28 smithi012 ceph-mon[31541]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-03-17T23:47:28.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:28 smithi012 ceph-mon[31541]: Updating smithi079:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-17T23:47:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:28.489+0000 7ffbc2600700 1 -- 172.21.15.12:0/422227858 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}) -- 0x7ffbbc19ad90 con 0x7ffba0039020 2024-03-17T23:47:29.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:29 smithi012 ceph-mon[31541]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:29.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:29 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:29.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:29 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:29.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:29 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:29.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:29 smithi012 ceph-mon[31541]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:47:30.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:30 smithi012 ceph-mon[31541]: Deploying cephadm binary to smithi191 2024-03-17T23:47:31.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:31 smithi012 ceph-mon[31541]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:33.488 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:33 smithi012 ceph-mon[31541]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:33.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:33.486+0000 7ffbb3e00700 1 -- 172.21.15.12:0/422227858 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7ffbbc19ad90 con 0x7ffba0039020 2024-03-17T23:47:33.489 INFO:teuthology.orchestra.run.smithi012.stdout:Added host 'smithi191' with addr '172.21.15.191' 2024-03-17T23:47:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:33.490+0000 7ffbc2600700 1 -- 172.21.15.12:0/422227858 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ffba0039020 msgr2=0x7ffba003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:33.490+0000 7ffbc2600700 1 --2- 172.21.15.12:0/422227858 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ffba0039020 0x7ffba003b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ffba8006fd0 tx=0x7ffba8006e40 comp rx=0 tx=0).stop 2024-03-17T23:47:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:33.490+0000 7ffbc2600700 1 -- 172.21.15.12:0/422227858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbbc102e60 msgr2=0x7ffbbc19f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:33.490+0000 7ffbc2600700 1 --2- 172.21.15.12:0/422227858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbbc102e60 0x7ffbbc19f6a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffbac006010 tx=0x7ffbac005030 comp rx=0 tx=0).stop 2024-03-17T23:47:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:33.490+0000 7ffbc2600700 1 -- 172.21.15.12:0/422227858 shutdown_connections 2024-03-17T23:47:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:33.490+0000 7ffbc2600700 1 --2- 172.21.15.12:0/422227858 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ffba0039020 0x7ffba003b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:33.490+0000 7ffbc2600700 1 --2- 172.21.15.12:0/422227858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbbc102e60 0x7ffbbc19f6a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:33.490+0000 7ffbc2600700 1 -- 172.21.15.12:0/422227858 >> 172.21.15.12:0/422227858 conn(0x7ffbbc077810 msgr2=0x7ffbbc108020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:33.490+0000 7ffbc2600700 1 -- 172.21.15.12:0/422227858 shutdown_connections 2024-03-17T23:47:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:33.490+0000 7ffbc2600700 1 -- 172.21.15.12:0/422227858 wait complete. 2024-03-17T23:47:34.278 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-17T23:47:34.731 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:47:34.865 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:34 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:34.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:34 smithi012 ceph-mon[31541]: Added host smithi191 2024-03-17T23:47:34.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:34 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:47:34.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:34 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:35.836 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:35 smithi012 ceph-mon[31541]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:35.837 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:36.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.573+0000 7f2ede800700 1 -- 172.21.15.12:0/3344196561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed8101f00 msgr2=0x7f2ed81022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.573+0000 7f2ede800700 1 --2- 172.21.15.12:0/3344196561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed8101f00 0x7f2ed81022e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009910 tx=0x7f2ec8009c20 comp rx=0 tx=0).stop 2024-03-17T23:47:36.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.573+0000 7f2ede800700 1 -- 172.21.15.12:0/3344196561 shutdown_connections 2024-03-17T23:47:36.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.573+0000 7f2ede800700 1 --2- 172.21.15.12:0/3344196561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed8101f00 0x7f2ed81022e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.573+0000 7f2ede800700 1 -- 172.21.15.12:0/3344196561 >> 172.21.15.12:0/3344196561 conn(0x7f2ed80fd810 msgr2=0x7f2ed80ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:36.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.574+0000 7f2ede800700 1 -- 172.21.15.12:0/3344196561 shutdown_connections 2024-03-17T23:47:36.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.574+0000 7f2ede800700 1 -- 172.21.15.12:0/3344196561 wait complete. 2024-03-17T23:47:36.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.575+0000 7f2ede800700 1 Processor -- start 2024-03-17T23:47:36.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.575+0000 7f2ede800700 1 -- start start 2024-03-17T23:47:36.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.576+0000 7f2ede800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed8101f00 0x7f2ed819d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.576+0000 7f2ede800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed810c2d0 con 0x7f2ed8101f00 2024-03-17T23:47:36.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.576+0000 7f2ed7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed8101f00 0x7f2ed819d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.576+0000 7f2ed7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed8101f00 0x7f2ed819d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34210/0 (socket says 172.21.15.12:34210) 2024-03-17T23:47:36.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.576+0000 7f2ed7400700 1 -- 172.21.15.12:0/1912684722 learned_addr learned my addr 172.21.15.12:0/1912684722 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:47:36.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.576+0000 7f2ed7400700 1 -- 172.21.15.12:0/1912684722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ed819db10 con 0x7f2ed8101f00 2024-03-17T23:47:36.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.577+0000 7f2ed7400700 1 --2- 172.21.15.12:0/1912684722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed8101f00 0x7f2ed819d5d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8006010 tx=0x7f2ec8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.577+0000 7f2ecfe00700 1 -- 172.21.15.12:0/1912684722 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ec8026070 con 0x7f2ed8101f00 2024-03-17T23:47:36.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.577+0000 7f2ecfe00700 1 -- 172.21.15.12:0/1912684722 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ec800bbd0 con 0x7f2ed8101f00 2024-03-17T23:47:36.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.577+0000 7f2ede800700 1 -- 172.21.15.12:0/1912684722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ed819de00 con 0x7f2ed8101f00 2024-03-17T23:47:36.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.577+0000 7f2ecfe00700 1 -- 172.21.15.12:0/1912684722 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ec8003910 con 0x7f2ed8101f00 2024-03-17T23:47:36.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.577+0000 7f2ede800700 1 -- 172.21.15.12:0/1912684722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ed8197870 con 0x7f2ed8101f00 2024-03-17T23:47:36.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.578+0000 7f2ede800700 1 -- 172.21.15.12:0/1912684722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ed81033e0 con 0x7f2ed8101f00 2024-03-17T23:47:36.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.579+0000 7f2ecfe00700 1 -- 172.21.15.12:0/1912684722 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f2ec8029070 con 0x7f2ed8101f00 2024-03-17T23:47:36.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.579+0000 7f2ecfe00700 1 --2- 172.21.15.12:0/1912684722 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2ebc039020 0x7f2ebc03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.579+0000 7f2ecfe00700 1 -- 172.21.15.12:0/1912684722 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2ec801e070 con 0x7f2ed8101f00 2024-03-17T23:47:36.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.580+0000 7f2ed6a00700 1 --2- 172.21.15.12:0/1912684722 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2ebc039020 0x7f2ebc03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.580+0000 7f2ed6a00700 1 --2- 172.21.15.12:0/1912684722 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2ebc039020 0x7f2ebc03b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006fd0 tx=0x7f2ec4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.585+0000 7f2ecfe00700 1 -- 172.21.15.12:0/1912684722 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2ec8013070 con 0x7f2ed8101f00 2024-03-17T23:47:36.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.818+0000 7f2ede800700 1 -- 172.21.15.12:0/1912684722 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2ed8198bd0 con 0x7f2ebc039020 2024-03-17T23:47:36.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.820+0000 7f2ecfe00700 1 -- 172.21.15.12:0/1912684722 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f2ed8198bd0 con 0x7f2ebc039020 2024-03-17T23:47:36.822 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:47:36.822 INFO:teuthology.orchestra.run.smithi012.stdout:[{"addr": "172.21.15.12", "hostname": "smithi012", "labels": [], "status": ""}, {"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}, {"addr": "172.21.15.191", "hostname": "smithi191", "labels": [], "status": ""}] 2024-03-17T23:47:36.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.823+0000 7f2ede800700 1 -- 172.21.15.12:0/1912684722 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2ebc039020 msgr2=0x7f2ebc03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.823+0000 7f2ede800700 1 --2- 172.21.15.12:0/1912684722 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2ebc039020 0x7f2ebc03b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006fd0 tx=0x7f2ec4006e40 comp rx=0 tx=0).stop 2024-03-17T23:47:36.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.823+0000 7f2ede800700 1 -- 172.21.15.12:0/1912684722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed8101f00 msgr2=0x7f2ed819d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.823+0000 7f2ede800700 1 --2- 172.21.15.12:0/1912684722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed8101f00 0x7f2ed819d5d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8006010 tx=0x7f2ec8005030 comp rx=0 tx=0).stop 2024-03-17T23:47:36.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.823+0000 7f2ede800700 1 -- 172.21.15.12:0/1912684722 shutdown_connections 2024-03-17T23:47:36.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.823+0000 7f2ede800700 1 --2- 172.21.15.12:0/1912684722 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2ebc039020 0x7f2ebc03b4e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.823+0000 7f2ede800700 1 --2- 172.21.15.12:0/1912684722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ed8101f00 0x7f2ed819d5d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.823+0000 7f2ede800700 1 -- 172.21.15.12:0/1912684722 >> 172.21.15.12:0/1912684722 conn(0x7f2ed80fd810 msgr2=0x7f2ed8104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:36.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.823+0000 7f2ede800700 1 -- 172.21.15.12:0/1912684722 shutdown_connections 2024-03-17T23:47:36.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:36.824+0000 7f2ede800700 1 -- 172.21.15.12:0/1912684722 wait complete. 2024-03-17T23:47:37.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:37 smithi012 ceph-mon[31541]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:37.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:37.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:37 smithi012 ceph-mon[31541]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:47:37.586 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-17T23:47:37.586 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-17T23:47:38.053 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:47:39.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:39 smithi012 ceph-mon[31541]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:39.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.936+0000 7fc55aa00700 1 -- 172.21.15.12:0/700082046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554100e00 msgr2=0x7fc5541011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:39.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.936+0000 7fc55aa00700 1 --2- 172.21.15.12:0/700082046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554100e00 0x7fc5541011e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc544009910 tx=0x7fc544009c20 comp rx=0 tx=0).stop 2024-03-17T23:47:39.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.937+0000 7fc55aa00700 1 -- 172.21.15.12:0/700082046 shutdown_connections 2024-03-17T23:47:39.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.937+0000 7fc55aa00700 1 --2- 172.21.15.12:0/700082046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554100e00 0x7fc5541011e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:39.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.937+0000 7fc55aa00700 1 -- 172.21.15.12:0/700082046 >> 172.21.15.12:0/700082046 conn(0x7fc554077810 msgr2=0x7fc5540feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:39.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.937+0000 7fc55aa00700 1 -- 172.21.15.12:0/700082046 shutdown_connections 2024-03-17T23:47:39.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.937+0000 7fc55aa00700 1 -- 172.21.15.12:0/700082046 wait complete. 2024-03-17T23:47:39.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.938+0000 7fc55aa00700 1 Processor -- start 2024-03-17T23:47:39.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.938+0000 7fc55aa00700 1 -- start start 2024-03-17T23:47:39.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.938+0000 7fc55aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554100e00 0x7fc55419d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:39.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.938+0000 7fc55aa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc55410b1d0 con 0x7fc554100e00 2024-03-17T23:47:39.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.939+0000 7fc553e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554100e00 0x7fc55419d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:39.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.939+0000 7fc553e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554100e00 0x7fc55419d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34234/0 (socket says 172.21.15.12:34234) 2024-03-17T23:47:39.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.939+0000 7fc553e00700 1 -- 172.21.15.12:0/3000369969 learned_addr learned my addr 172.21.15.12:0/3000369969 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:47:39.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.939+0000 7fc553e00700 1 -- 172.21.15.12:0/3000369969 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc55419db00 con 0x7fc554100e00 2024-03-17T23:47:39.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.939+0000 7fc553e00700 1 --2- 172.21.15.12:0/3000369969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554100e00 0x7fc55419d5c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc544006010 tx=0x7fc544005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:39.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.940+0000 7fc551600700 1 -- 172.21.15.12:0/3000369969 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc544026070 con 0x7fc554100e00 2024-03-17T23:47:39.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.940+0000 7fc551600700 1 -- 172.21.15.12:0/3000369969 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc54400bbd0 con 0x7fc554100e00 2024-03-17T23:47:39.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.940+0000 7fc55aa00700 1 -- 172.21.15.12:0/3000369969 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc55419ddf0 con 0x7fc554100e00 2024-03-17T23:47:39.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.940+0000 7fc551600700 1 -- 172.21.15.12:0/3000369969 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc544003910 con 0x7fc554100e00 2024-03-17T23:47:39.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.940+0000 7fc55aa00700 1 -- 172.21.15.12:0/3000369969 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc554197860 con 0x7fc554100e00 2024-03-17T23:47:39.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.941+0000 7fc551600700 1 -- 172.21.15.12:0/3000369969 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fc544029070 con 0x7fc554100e00 2024-03-17T23:47:39.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.941+0000 7fc55aa00700 1 -- 172.21.15.12:0/3000369969 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5541022e0 con 0x7fc554100e00 2024-03-17T23:47:39.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.942+0000 7fc551600700 1 --2- 172.21.15.12:0/3000369969 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc538039020 0x7fc53803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:39.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.942+0000 7fc551600700 1 -- 172.21.15.12:0/3000369969 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc54401e070 con 0x7fc554100e00 2024-03-17T23:47:39.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.942+0000 7fc54be00700 1 --2- 172.21.15.12:0/3000369969 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc538039020 0x7fc53803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:39.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.943+0000 7fc54be00700 1 --2- 172.21.15.12:0/3000369969 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc538039020 0x7fc53803b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc53c006fd0 tx=0x7fc53c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:39.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:39.947+0000 7fc551600700 1 -- 172.21.15.12:0/3000369969 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc544013070 con 0x7fc554100e00 2024-03-17T23:47:40.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:40.183+0000 7fc55aa00700 1 -- 172.21.15.12:0/3000369969 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fc554198bc0 con 0x7fc554100e00 2024-03-17T23:47:40.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:40 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:40.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:40 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:40.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:40 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:40.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:40 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:40.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:40 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:47:40.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:40 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:47:40.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:40 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:47:40.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:40 smithi012 ceph-mon[31541]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-17T23:47:40.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:40 smithi012 ceph-mon[31541]: Updating smithi191:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:47:40.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:40 smithi012 ceph-mon[31541]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-03-17T23:47:41.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:41.181+0000 7fc551600700 1 -- 172.21.15.12:0/3000369969 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fc54400fe20 con 0x7fc554100e00 2024-03-17T23:47:41.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:41.185+0000 7fc55aa00700 1 -- 172.21.15.12:0/3000369969 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc538039020 msgr2=0x7fc53803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:41.185+0000 7fc55aa00700 1 --2- 172.21.15.12:0/3000369969 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc538039020 0x7fc53803b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc53c006fd0 tx=0x7fc53c006e40 comp rx=0 tx=0).stop 2024-03-17T23:47:41.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:41.185+0000 7fc55aa00700 1 -- 172.21.15.12:0/3000369969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554100e00 msgr2=0x7fc55419d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:41.185+0000 7fc55aa00700 1 --2- 172.21.15.12:0/3000369969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554100e00 0x7fc55419d5c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc544006010 tx=0x7fc544005030 comp rx=0 tx=0).stop 2024-03-17T23:47:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:41.186+0000 7fc55aa00700 1 -- 172.21.15.12:0/3000369969 shutdown_connections 2024-03-17T23:47:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:41.186+0000 7fc55aa00700 1 --2- 172.21.15.12:0/3000369969 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc538039020 0x7fc53803b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:41.186+0000 7fc55aa00700 1 --2- 172.21.15.12:0/3000369969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc554100e00 0x7fc55419d5c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:41.186+0000 7fc55aa00700 1 -- 172.21.15.12:0/3000369969 >> 172.21.15.12:0/3000369969 conn(0x7fc554077810 msgr2=0x7fc5540696c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:41.186+0000 7fc55aa00700 1 -- 172.21.15.12:0/3000369969 shutdown_connections 2024-03-17T23:47:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:47:41.186+0000 7fc55aa00700 1 -- 172.21.15.12:0/3000369969 wait complete. 2024-03-17T23:47:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2024-03-17T23:47:41.542 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:41 smithi012 ceph-mon[31541]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:41.543 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:41 smithi012 ceph-mon[31541]: Updating smithi191:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-17T23:47:41.543 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:41 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3000369969' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-17T23:47:41.543 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:41 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:41.543 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:41 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:41.543 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:41 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:41.876 INFO:tasks.cephadm:Adding mon.a on smithi012 2024-03-17T23:47:41.876 INFO:tasks.cephadm:Adding mon.b on smithi079 2024-03-17T23:47:41.876 INFO:tasks.cephadm:Adding mon.c on smithi191 2024-03-17T23:47:41.877 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi012:172.21.15.12=a;smithi079:172.21.15.79=b;smithi191:172.21.15.191=c' 2024-03-17T23:47:42.115 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:47:42.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:42 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3000369969' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-17T23:47:42.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:42 smithi012 ceph-mon[31541]: osdmap e4: 0 total, 0 up, 0 in 2024-03-17T23:47:43.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.627+0000 7f067f63a700 1 -- 172.21.15.191:0/848973005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0678106d10 msgr2=0x7f06781070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.627+0000 7f067f63a700 1 --2- 172.21.15.191:0/848973005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0678106d10 0x7f06781070f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f067000b0a0 tx=0x7f067000b3b0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.627+0000 7f067f63a700 1 -- 172.21.15.191:0/848973005 shutdown_connections 2024-03-17T23:47:43.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.627+0000 7f067f63a700 1 --2- 172.21.15.191:0/848973005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0678106d10 0x7f06781070f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.627+0000 7f067f63a700 1 -- 172.21.15.191:0/848973005 >> 172.21.15.191:0/848973005 conn(0x7f06780fbe50 msgr2=0x7f06780fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:43.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.627+0000 7f067f63a700 1 -- 172.21.15.191:0/848973005 shutdown_connections 2024-03-17T23:47:43.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.627+0000 7f067f63a700 1 -- 172.21.15.191:0/848973005 wait complete. 2024-03-17T23:47:43.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.628+0000 7f067f63a700 1 Processor -- start 2024-03-17T23:47:43.630 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.628+0000 7f067f63a700 1 -- start start 2024-03-17T23:47:43.630 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.628+0000 7f067f63a700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0678106d10 0x7f067819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.630 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.628+0000 7f067f63a700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f067810c470 con 0x7f0678106d10 2024-03-17T23:47:43.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.628+0000 7f067d3d6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0678106d10 0x7f067819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.629+0000 7f067d3d6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0678106d10 0x7f067819d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.191:42910/0 (socket says 172.21.15.191:42910) 2024-03-17T23:47:43.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.629+0000 7f067d3d6700 1 -- 172.21.15.191:0/4248990731 learned_addr learned my addr 172.21.15.191:0/4248990731 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-17T23:47:43.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.629+0000 7f067d3d6700 1 -- 172.21.15.191:0/4248990731 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067819dab0 con 0x7f0678106d10 2024-03-17T23:47:43.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.629+0000 7f067d3d6700 1 --2- 172.21.15.191:0/4248990731 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0678106d10 0x7f067819d570 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f06700062a0 tx=0x7f0670003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:43.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.630+0000 7f066e7fc700 1 -- 172.21.15.191:0/4248990731 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0670017040 con 0x7f0678106d10 2024-03-17T23:47:43.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.630+0000 7f067f63a700 1 -- 172.21.15.191:0/4248990731 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067819dd40 con 0x7f0678106d10 2024-03-17T23:47:43.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.630+0000 7f067f63a700 1 -- 172.21.15.191:0/4248990731 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0678197640 con 0x7f0678106d10 2024-03-17T23:47:43.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.630+0000 7f066e7fc700 1 -- 172.21.15.191:0/4248990731 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0670022070 con 0x7f0678106d10 2024-03-17T23:47:43.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.630+0000 7f066e7fc700 1 -- 172.21.15.191:0/4248990731 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0670009410 con 0x7f0678106d10 2024-03-17T23:47:43.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.631+0000 7f067f63a700 1 -- 172.21.15.191:0/4248990731 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f065c005330 con 0x7f0678106d10 2024-03-17T23:47:43.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.632+0000 7f066e7fc700 1 -- 172.21.15.191:0/4248990731 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f067000bb90 con 0x7f0678106d10 2024-03-17T23:47:43.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.632+0000 7f066e7fc700 1 --2- 172.21.15.191:0/4248990731 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0664039160 0x7f066403b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.632+0000 7f066e7fc700 1 -- 172.21.15.191:0/4248990731 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f067000f050 con 0x7f0678106d10 2024-03-17T23:47:43.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.633+0000 7f067cbd5700 1 --2- 172.21.15.191:0/4248990731 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0664039160 0x7f066403b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.633+0000 7f067cbd5700 1 --2- 172.21.15.191:0/4248990731 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0664039160 0x7f066403b620 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0668006fd0 tx=0x7f0668006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:43.648 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.647+0000 7f066e7fc700 1 -- 172.21.15.191:0/4248990731 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0670007e20 con 0x7f0678106d10 2024-03-17T23:47:43.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:43 smithi012 ceph-mon[31541]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:43.853 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.852+0000 7f067f63a700 1 -- 172.21.15.191:0/4248990731 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi012:172.21.15.12=a;smithi079:172.21.15.79=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}) -- 0x7f065c000c80 con 0x7f0664039160 2024-03-17T23:47:43.868 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.867+0000 7f066e7fc700 1 -- 172.21.15.191:0/4248990731 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f065c000c80 con 0x7f0664039160 2024-03-17T23:47:43.869 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mon update... 2024-03-17T23:47:43.871 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.870+0000 7f067f63a700 1 -- 172.21.15.191:0/4248990731 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0664039160 msgr2=0x7f066403b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.871 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.870+0000 7f067f63a700 1 --2- 172.21.15.191:0/4248990731 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0664039160 0x7f066403b620 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0668006fd0 tx=0x7f0668006e40 comp rx=0 tx=0).stop 2024-03-17T23:47:43.871 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.870+0000 7f067f63a700 1 -- 172.21.15.191:0/4248990731 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0678106d10 msgr2=0x7f067819d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.871 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.870+0000 7f067f63a700 1 --2- 172.21.15.191:0/4248990731 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0678106d10 0x7f067819d570 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f06700062a0 tx=0x7f0670003ce0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.871 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.870+0000 7f067f63a700 1 -- 172.21.15.191:0/4248990731 shutdown_connections 2024-03-17T23:47:43.871 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.870+0000 7f067f63a700 1 --2- 172.21.15.191:0/4248990731 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0664039160 0x7f066403b620 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.871 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.870+0000 7f067f63a700 1 --2- 172.21.15.191:0/4248990731 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0678106d10 0x7f067819d570 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.871 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.870+0000 7f067f63a700 1 -- 172.21.15.191:0/4248990731 >> 172.21.15.191:0/4248990731 conn(0x7f06780fbe50 msgr2=0x7f06781097a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:43.871 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.870+0000 7f067f63a700 1 -- 172.21.15.191:0/4248990731 shutdown_connections 2024-03-17T23:47:43.872 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:43.870+0000 7f067f63a700 1 -- 172.21.15.191:0/4248990731 wait complete. 2024-03-17T23:47:44.522 DEBUG:teuthology.orchestra.run.smithi079:mon.b> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.b.service 2024-03-17T23:47:44.524 DEBUG:teuthology.orchestra.run.smithi191:mon.c> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.c.service 2024-03-17T23:47:44.527 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-17T23:47:44.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-17T23:47:45.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:44 smithi012 ceph-mon[31541]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi012:172.21.15.12=a;smithi079:172.21.15.79=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:47:45.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:44 smithi012 ceph-mon[31541]: Saving service mon spec with placement smithi012:172.21.15.12=a;smithi079:172.21.15.79=b;smithi191:172.21.15.191=c;count:3 2024-03-17T23:47:45.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:45.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:47:45.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:47:45.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:47:45.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:45.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-17T23:47:45.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:47:45.650 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:47:46.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:45 smithi012 ceph-mon[31541]: Deploying daemon mon.c on smithi191 2024-03-17T23:47:46.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:45 smithi012 ceph-mon[31541]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:48.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:47 smithi012 ceph-mon[31541]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:48.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.580+0000 7f171d69d700 1 -- 172.21.15.191:0/3641263122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1718101f00 msgr2=0x7f17181022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.580+0000 7f171d69d700 1 --2- 172.21.15.191:0/3641263122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1718101f00 0x7f17181022e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1708009910 tx=0x7f1708009c20 comp rx=0 tx=0).stop 2024-03-17T23:47:48.583 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.580+0000 7f171d69d700 1 -- 172.21.15.191:0/3641263122 shutdown_connections 2024-03-17T23:47:48.583 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.580+0000 7f171d69d700 1 --2- 172.21.15.191:0/3641263122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1718101f00 0x7f17181022e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.583 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.580+0000 7f171d69d700 1 -- 172.21.15.191:0/3641263122 >> 172.21.15.191:0/3641263122 conn(0x7f17180fd790 msgr2=0x7f17180ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:48.583 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.580+0000 7f171d69d700 1 -- 172.21.15.191:0/3641263122 shutdown_connections 2024-03-17T23:47:48.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.580+0000 7f171d69d700 1 -- 172.21.15.191:0/3641263122 wait complete. 2024-03-17T23:47:48.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.581+0000 7f171d69d700 1 Processor -- start 2024-03-17T23:47:48.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.581+0000 7f171d69d700 1 -- start start 2024-03-17T23:47:48.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.581+0000 7f171d69d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1718101f00 0x7f171819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.581+0000 7f171d69d700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f171810c2d0 con 0x7f1718101f00 2024-03-17T23:47:48.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.582+0000 7f1716ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1718101f00 0x7f171819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.582+0000 7f1716ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1718101f00 0x7f171819d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.191:44396/0 (socket says 172.21.15.191:44396) 2024-03-17T23:47:48.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.582+0000 7f1716ffd700 1 -- 172.21.15.191:0/1729891814 learned_addr learned my addr 172.21.15.191:0/1729891814 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-17T23:47:48.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.583+0000 7f1716ffd700 1 -- 172.21.15.191:0/1729891814 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171819d980 con 0x7f1718101f00 2024-03-17T23:47:48.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.583+0000 7f1716ffd700 1 --2- 172.21.15.191:0/1729891814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1718101f00 0x7f171819d440 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1708005850 tx=0x7f1708005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:48.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.583+0000 7f1714ff9700 1 -- 172.21.15.191:0/1729891814 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1708026070 con 0x7f1718101f00 2024-03-17T23:47:48.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.583+0000 7f171d69d700 1 -- 172.21.15.191:0/1729891814 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f171819dc70 con 0x7f1718101f00 2024-03-17T23:47:48.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.583+0000 7f1714ff9700 1 -- 172.21.15.191:0/1729891814 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f170800bbd0 con 0x7f1718101f00 2024-03-17T23:47:48.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.583+0000 7f1714ff9700 1 -- 172.21.15.191:0/1729891814 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1708003970 con 0x7f1718101f00 2024-03-17T23:47:48.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.583+0000 7f171d69d700 1 -- 172.21.15.191:0/1729891814 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1718197600 con 0x7f1718101f00 2024-03-17T23:47:48.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.584+0000 7f171d69d700 1 -- 172.21.15.191:0/1729891814 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17181033e0 con 0x7f1718101f00 2024-03-17T23:47:48.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.585+0000 7f1714ff9700 1 -- 172.21.15.191:0/1729891814 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f1708029070 con 0x7f1718101f00 2024-03-17T23:47:48.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.585+0000 7f1714ff9700 1 --2- 172.21.15.191:0/1729891814 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1704038fc0 0x7f170403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.585+0000 7f1714ff9700 1 -- 172.21.15.191:0/1729891814 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f170801e070 con 0x7f1718101f00 2024-03-17T23:47:48.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.586+0000 7f170e7fc700 1 --2- 172.21.15.191:0/1729891814 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1704038fc0 0x7f170403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.587+0000 7f170e7fc700 1 --2- 172.21.15.191:0/1729891814 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1704038fc0 0x7f170403b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1700006fd0 tx=0x7f1700006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:48.603 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.602+0000 7f1714ff9700 1 -- 172.21.15.191:0/1729891814 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1708030020 con 0x7f1718101f00 2024-03-17T23:47:48.902 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.900+0000 7f171d69d700 1 -- 172.21.15.191:0/1729891814 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f1718198b30 con 0x7f1718101f00 2024-03-17T23:47:48.902 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.901+0000 7f1714ff9700 1 -- 172.21.15.191:0/1729891814 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f1708013070 con 0x7f1718101f00 2024-03-17T23:47:48.902 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:47:48.902 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":1,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","modified":"2024-03-17T23:45:34.840429Z","created":"2024-03-17T23:45:34.840429Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-17T23:47:48.905 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.903+0000 7f171d69d700 1 -- 172.21.15.191:0/1729891814 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1704038fc0 msgr2=0x7f170403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.905 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.903+0000 7f171d69d700 1 --2- 172.21.15.191:0/1729891814 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1704038fc0 0x7f170403b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1700006fd0 tx=0x7f1700006e40 comp rx=0 tx=0).stop 2024-03-17T23:47:48.906 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.903+0000 7f171d69d700 1 -- 172.21.15.191:0/1729891814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1718101f00 msgr2=0x7f171819d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.906 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.903+0000 7f171d69d700 1 --2- 172.21.15.191:0/1729891814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1718101f00 0x7f171819d440 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1708005850 tx=0x7f1708005030 comp rx=0 tx=0).stop 2024-03-17T23:47:48.906 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.903+0000 7f171d69d700 1 -- 172.21.15.191:0/1729891814 shutdown_connections 2024-03-17T23:47:48.906 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.903+0000 7f171d69d700 1 --2- 172.21.15.191:0/1729891814 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1704038fc0 0x7f170403b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.906 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.903+0000 7f171d69d700 1 --2- 172.21.15.191:0/1729891814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1718101f00 0x7f171819d440 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.906 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.903+0000 7f171d69d700 1 -- 172.21.15.191:0/1729891814 >> 172.21.15.191:0/1729891814 conn(0x7f17180fd790 msgr2=0x7f17181038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:48.906 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.903+0000 7f171d69d700 1 -- 172.21.15.191:0/1729891814 shutdown_connections 2024-03-17T23:47:48.906 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:48.903+0000 7f171d69d700 1 -- 172.21.15.191:0/1729891814 wait complete. 2024-03-17T23:47:48.907 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 1 2024-03-17T23:47:50.657 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-17T23:47:50.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-17T23:47:50.903 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-17T23:47:54.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:47:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:54.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:47:54.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: mon.a calling monitor election 2024-03-17T23:47:54.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: mon.c calling monitor election 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: monmap epoch 2 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: last_changed 2024-03-17T23:47:48.976701+0000 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: created 2024-03-17T23:45:34.840429+0000 2024-03-17T23:47:54.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: min_mon_release 19 (squid) 2024-03-17T23:47:54.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: election_strategy: 1 2024-03-17T23:47:54.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-03-17T23:47:54.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-03-17T23:47:54.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: fsmap 2024-03-17T23:47:54.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: osdmap e4: 0 total, 0 up, 0 in 2024-03-17T23:47:54.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: mgrmap e13: x(active, since 60s) 2024-03-17T23:47:54.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: overall HEALTH_OK 2024-03-17T23:47:54.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:54.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:55.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:47:55 smithi191 ceph-mon[36708]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:55.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:47:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:55.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:47:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-17T23:47:55.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:47:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:47:55.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:47:55 smithi191 ceph-mon[36708]: Deploying daemon mon.b on smithi079 2024-03-17T23:47:55.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:47:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:47:55.378 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.378+0000 7fe740da1700 1 -- 172.21.15.191:0/3000007211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7240057d0 msgr2=0x7fe724007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.378 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.378+0000 7fe740da1700 1 --2- 172.21.15.191:0/3000007211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7240057d0 0x7fe724007c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe73c100bd0 tx=0x7fe72c00feb0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.378 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.378+0000 7fe740da1700 1 -- 172.21.15.191:0/3000007211 shutdown_connections 2024-03-17T23:47:55.378 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.378+0000 7fe740da1700 1 --2- 172.21.15.191:0/3000007211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7240057d0 0x7fe724007c60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.378 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.378+0000 7fe740da1700 1 -- 172.21.15.191:0/3000007211 >> 172.21.15.191:0/3000007211 conn(0x7fe73c0fc770 msgr2=0x7fe73c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:55.378 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.378+0000 7fe740da1700 1 -- 172.21.15.191:0/3000007211 shutdown_connections 2024-03-17T23:47:55.379 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.378+0000 7fe740da1700 1 -- 172.21.15.191:0/3000007211 wait complete. 2024-03-17T23:47:55.379 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.379+0000 7fe740da1700 1 Processor -- start 2024-03-17T23:47:55.379 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.380+0000 7fe740da1700 1 -- start start 2024-03-17T23:47:55.380 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.380+0000 7fe740da1700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7240057d0 0x7fe73c19b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.380 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.380+0000 7fe740da1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe73c19b930 0x7fe73c1955b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.380 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.380+0000 7fe740da1700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe73c107d00 con 0x7fe7240057d0 2024-03-17T23:47:55.381 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.380+0000 7fe740da1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe73c107b80 con 0x7fe73c19b930 2024-03-17T23:47:55.381 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.380+0000 7fe739d9b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe73c19b930 0x7fe73c1955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.381 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.380+0000 7fe739d9b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe73c19b930 0x7fe73c1955b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:49750/0 (socket says 172.21.15.191:49750) 2024-03-17T23:47:55.381 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.380+0000 7fe739d9b700 1 -- 172.21.15.191:0/2025977928 learned_addr learned my addr 172.21.15.191:0/2025977928 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-17T23:47:55.381 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.380+0000 7fe739d9b700 1 -- 172.21.15.191:0/2025977928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe73c19b930 msgr2=0x7fe73c1955b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-17T23:47:55.381 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.380+0000 7fe739d9b700 1 -- 172.21.15.191:0/2025977928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe73c19b930 msgr2=0x7fe73c1955b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-17T23:47:55.381 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.380+0000 7fe739d9b700 1 --2- 172.21.15.191:0/2025977928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe73c19b930 0x7fe73c1955b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-17T23:47:55.381 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.380+0000 7fe739d9b700 1 --2- 172.21.15.191:0/2025977928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe73c19b930 0x7fe73c1955b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-17T23:47:55.381 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.381+0000 7fe73a59c700 1 --2- 172.21.15.191:0/2025977928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7240057d0 0x7fe73c19b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.382 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.381+0000 7fe73a59c700 1 -- 172.21.15.191:0/2025977928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe73c19b930 msgr2=0x7fe73c1955b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-17T23:47:55.382 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.381+0000 7fe73a59c700 1 --2- 172.21.15.191:0/2025977928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe73c19b930 0x7fe73c1955b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.382 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.381+0000 7fe73a59c700 1 -- 172.21.15.191:0/2025977928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe73c19bf70 con 0x7fe7240057d0 2024-03-17T23:47:55.382 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.382+0000 7fe73a59c700 1 --2- 172.21.15.191:0/2025977928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7240057d0 0x7fe73c19b3f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe72c010010 tx=0x7fe72c0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:55.383 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.382+0000 7fe72b7fe700 1 -- 172.21.15.191:0/2025977928 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe72c004b20 con 0x7fe7240057d0 2024-03-17T23:47:55.383 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.382+0000 7fe72b7fe700 1 -- 172.21.15.191:0/2025977928 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe72c013e40 con 0x7fe7240057d0 2024-03-17T23:47:55.383 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.382+0000 7fe740da1700 1 -- 172.21.15.191:0/2025977928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe73c195af0 con 0x7fe7240057d0 2024-03-17T23:47:55.383 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.382+0000 7fe72b7fe700 1 -- 172.21.15.191:0/2025977928 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe72c011b60 con 0x7fe7240057d0 2024-03-17T23:47:55.383 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.383+0000 7fe740da1700 1 -- 172.21.15.191:0/2025977928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe73c196080 con 0x7fe7240057d0 2024-03-17T23:47:55.383 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.384+0000 7fe740da1700 1 -- 172.21.15.191:0/2025977928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe73c1963f0 con 0x7fe7240057d0 2024-03-17T23:47:55.384 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.384+0000 7fe72b7fe700 1 -- 172.21.15.191:0/2025977928 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fe72c023070 con 0x7fe7240057d0 2024-03-17T23:47:55.384 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.384+0000 7fe72b7fe700 1 --2- 172.21.15.191:0/2025977928 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe71c038db0 0x7fe71c03b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.384 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.384+0000 7fe72b7fe700 1 -- 172.21.15.191:0/2025977928 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe72c020050 con 0x7fe7240057d0 2024-03-17T23:47:55.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.385+0000 7fe739d9b700 1 --2- 172.21.15.191:0/2025977928 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe71c038db0 0x7fe71c03b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.386+0000 7fe739d9b700 1 --2- 172.21.15.191:0/2025977928 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe71c038db0 0x7fe71c03b270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe730009180 tx=0x7fe730008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:55.399 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.399+0000 7fe72b7fe700 1 -- 172.21.15.191:0/2025977928 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe72c028020 con 0x7fe7240057d0 2024-03-17T23:47:55.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:55 smithi012 ceph-mon[31541]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:55.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:47:55.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-17T23:47:55.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:47:55.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:55 smithi012 ceph-mon[31541]: Deploying daemon mon.b on smithi079 2024-03-17T23:47:55.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:47:55.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.674+0000 7fe740da1700 1 -- 172.21.15.191:0/2025977928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe73c196ae0 con 0x7fe7240057d0 2024-03-17T23:47:55.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.675+0000 7fe72b7fe700 1 -- 172.21.15.191:0/2025977928 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fe72c011dc0 con 0x7fe7240057d0 2024-03-17T23:47:55.675 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:47:55.675 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":2,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","modified":"2024-03-17T23:47:48.976701Z","created":"2024-03-17T23:45:34.840429Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-17T23:47:55.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.677+0000 7fe740da1700 1 -- 172.21.15.191:0/2025977928 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe71c038db0 msgr2=0x7fe71c03b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.677+0000 7fe740da1700 1 --2- 172.21.15.191:0/2025977928 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe71c038db0 0x7fe71c03b270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe730009180 tx=0x7fe730008040 comp rx=0 tx=0).stop 2024-03-17T23:47:55.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.677+0000 7fe740da1700 1 -- 172.21.15.191:0/2025977928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7240057d0 msgr2=0x7fe73c19b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.677+0000 7fe740da1700 1 --2- 172.21.15.191:0/2025977928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7240057d0 0x7fe73c19b3f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe72c010010 tx=0x7fe72c0055a0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.677+0000 7fe740da1700 1 -- 172.21.15.191:0/2025977928 shutdown_connections 2024-03-17T23:47:55.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.677+0000 7fe740da1700 1 --2- 172.21.15.191:0/2025977928 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe71c038db0 0x7fe71c03b270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.677+0000 7fe740da1700 1 --2- 172.21.15.191:0/2025977928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7240057d0 0x7fe73c19b3f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.677+0000 7fe740da1700 1 --2- 172.21.15.191:0/2025977928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe73c19b930 0x7fe73c1955b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.677+0000 7fe740da1700 1 -- 172.21.15.191:0/2025977928 >> 172.21.15.191:0/2025977928 conn(0x7fe73c0fc770 msgr2=0x7fe73c0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:55.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.677+0000 7fe740da1700 1 -- 172.21.15.191:0/2025977928 shutdown_connections 2024-03-17T23:47:55.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:47:55.677+0000 7fe740da1700 1 -- 172.21.15.191:0/2025977928 wait complete. 2024-03-17T23:47:55.678 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 2 2024-03-17T23:47:56.241 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:47:56 smithi191 ceph-mon[36708]: from='client.? 172.21.15.191:0/2025977928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-17T23:47:56.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:56 smithi012 ceph-mon[31541]: from='client.? 172.21.15.191:0/2025977928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-17T23:47:56.419 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:47:55 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:47:55.977+0000 7fb970000700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-17T23:47:57.243 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-17T23:47:57.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-17T23:47:57.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:47:57 smithi012 ceph-mon[31541]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:57.487 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-17T23:47:57.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:47:57 smithi191 ceph-mon[36708]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:47:58.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:47:58 smithi079 ceph-mon[36241]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-17T23:47:58.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:47:58 smithi079 bash[36221]: 3d8775146265547bc26009cb3e96dc7268e1e7013a112aa12c4f69591cc18137 2024-03-17T23:47:58.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:47:58 smithi079 systemd[1]: Started Ceph mon.b for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:48:03.348 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.345+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1581861851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cf00035c0 msgr2=0x7f7cf0005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:03.348 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.345+0000 7f7d0c63f700 1 --2- 172.21.15.191:0/1581861851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cf00035c0 0x7f7cf0005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7d04101ce0 tx=0x7f7cfc00fe60 comp rx=0 tx=0).stop 2024-03-17T23:48:03.348 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.346+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1581861851 shutdown_connections 2024-03-17T23:48:03.348 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.346+0000 7f7d0c63f700 1 --2- 172.21.15.191:0/1581861851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cf00035c0 0x7f7cf0005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.348 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.346+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1581861851 >> 172.21.15.191:0/1581861851 conn(0x7f7d040fd8c0 msgr2=0x7f7d040ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:03.348 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.346+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1581861851 shutdown_connections 2024-03-17T23:48:03.348 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.346+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1581861851 wait complete. 2024-03-17T23:48:03.348 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.346+0000 7f7d0c63f700 1 Processor -- start 2024-03-17T23:48:03.348 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d0c63f700 1 -- start start 2024-03-17T23:48:03.348 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d0c63f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cf00035c0 0x7f7d0419f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:03.348 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d0c63f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d0419fdf0 0x7f7d04199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d0c63f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d04199fb0 0x7f7d0419a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d0c63f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7d04075660 con 0x7f7cf00035c0 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d0c63f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7d040754e0 con 0x7f7d04199fb0 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d0c63f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7d040757e0 con 0x7f7d0419fdf0 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d09bda700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d0419fdf0 0x7f7d04199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d09bda700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d0419fdf0 0x7f7d04199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:49344/0 (socket says 172.21.15.191:49344) 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d09bda700 1 -- 172.21.15.191:0/1560509248 learned_addr learned my addr 172.21.15.191:0/1560509248 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d09bda700 1 -- 172.21.15.191:0/1560509248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d0419fdf0 msgr2=0x7f7d04199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d09bda700 1 -- 172.21.15.191:0/1560509248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d0419fdf0 msgr2=0x7f7d04199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d09bda700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d0419fdf0 0x7f7d04199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d0abdc700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d04199fb0 0x7f7d0419a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d09bda700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d0419fdf0 0x7f7d04199a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d0a3db700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cf00035c0 0x7f7d0419f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:03.349 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.347+0000 7f7d0abdc700 1 -- 172.21.15.191:0/1560509248 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d04199fb0 msgr2=0x7f7d0419a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-17T23:48:03.350 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.348+0000 7f7d0abdc700 1 -- 172.21.15.191:0/1560509248 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d04199fb0 msgr2=0x7f7d0419a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-17T23:48:03.350 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.348+0000 7f7d0abdc700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d04199fb0 0x7f7d0419a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-17T23:48:03.350 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.348+0000 7f7d0abdc700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d04199fb0 0x7f7d0419a460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-17T23:48:03.351 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.348+0000 7f7d0a3db700 1 -- 172.21.15.191:0/1560509248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d0419fdf0 msgr2=0x7f7d04199a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-17T23:48:03.351 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.348+0000 7f7d0a3db700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d0419fdf0 0x7f7d04199a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.351 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.348+0000 7f7d0a3db700 1 -- 172.21.15.191:0/1560509248 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d04199fb0 msgr2=0x7f7d0419a460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-17T23:48:03.351 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.348+0000 7f7d0a3db700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d04199fb0 0x7f7d0419a460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.351 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.348+0000 7f7d0a3db700 1 -- 172.21.15.191:0/1560509248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d041a6780 con 0x7f7cf00035c0 2024-03-17T23:48:03.352 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.349+0000 7f7d0a3db700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cf00035c0 0x7f7d0419f8b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7d04101ce0 tx=0x7f7cfc005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:03.352 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.349+0000 7f7cfb7fe700 1 -- 172.21.15.191:0/1560509248 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc02b030 con 0x7f7cf00035c0 2024-03-17T23:48:03.352 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.349+0000 7f7cfb7fe700 1 -- 172.21.15.191:0/1560509248 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cfc012e30 con 0x7f7cf00035c0 2024-03-17T23:48:03.352 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.349+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1560509248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d041a6950 con 0x7f7cf00035c0 2024-03-17T23:48:03.352 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.349+0000 7f7cfb7fe700 1 -- 172.21.15.191:0/1560509248 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc022040 con 0x7f7cf00035c0 2024-03-17T23:48:03.352 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.349+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1560509248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d041a6e90 con 0x7f7cf00035c0 2024-03-17T23:48:03.352 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.350+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1560509248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d04063900 con 0x7f7cf00035c0 2024-03-17T23:48:03.352 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.350+0000 7f7cfb7fe700 1 -- 172.21.15.191:0/1560509248 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f7cfc02e070 con 0x7f7cf00035c0 2024-03-17T23:48:03.352 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.351+0000 7f7cfb7fe700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7ce8039450 0x7f7ce803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:03.352 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.351+0000 7f7cfb7fe700 1 -- 172.21.15.191:0/1560509248 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7cfc04e530 con 0x7f7cf00035c0 2024-03-17T23:48:03.352 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.351+0000 7f7d09bda700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7ce8039450 0x7f7ce803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:03.353 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.353+0000 7f7d09bda700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7ce8039450 0x7f7ce803b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009180 tx=0x7f7cf4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:03.368 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.368+0000 7f7cfb7fe700 1 -- 172.21.15.191:0/1560509248 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7cfc010630 con 0x7f7cf00035c0 2024-03-17T23:48:03.369 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:03.369 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: mon.c calling monitor election 2024-03-17T23:48:03.369 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:48:03.369 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: mon.a calling monitor election 2024-03-17T23:48:03.369 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:03.369 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:48:03.369 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:03.369 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:03.369 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:03.369 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:03.369 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:03.369 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:03.370 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:03.370 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-17T23:48:03.370 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: monmap epoch 3 2024-03-17T23:48:03.370 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:48:03.370 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: last_changed 2024-03-17T23:47:58.038625+0000 2024-03-17T23:48:03.370 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: created 2024-03-17T23:45:34.840429+0000 2024-03-17T23:48:03.370 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: min_mon_release 19 (squid) 2024-03-17T23:48:03.370 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: election_strategy: 1 2024-03-17T23:48:03.370 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-03-17T23:48:03.370 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-03-17T23:48:03.370 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: 2: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.b 2024-03-17T23:48:03.370 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: fsmap 2024-03-17T23:48:03.370 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: osdmap e4: 0 total, 0 up, 0 in 2024-03-17T23:48:03.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: mgrmap e13: x(active, since 69s) 2024-03-17T23:48:03.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-17T23:48:03.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-17T23:48:03.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-17T23:48:03.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: mon.b (rank 2) addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] is down (out of quorum) 2024-03-17T23:48:03.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:03.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:03.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:03.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: mon.c calling monitor election 2024-03-17T23:48:03.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:48:03.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: mon.a calling monitor election 2024-03-17T23:48:03.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:03.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:48:03.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:03.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:03.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:03.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:03.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: monmap epoch 3 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: last_changed 2024-03-17T23:47:58.038625+0000 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: created 2024-03-17T23:45:34.840429+0000 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: min_mon_release 19 (squid) 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: election_strategy: 1 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: 2: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.b 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: fsmap 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: osdmap e4: 0 total, 0 up, 0 in 2024-03-17T23:48:03.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: mgrmap e13: x(active, since 69s) 2024-03-17T23:48:03.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-17T23:48:03.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-17T23:48:03.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-17T23:48:03.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: mon.b (rank 2) addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] is down (out of quorum) 2024-03-17T23:48:03.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:03.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:03.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.662+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1560509248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7d0402db70 con 0x7f7cf00035c0 2024-03-17T23:48:03.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.663+0000 7f7cfb7fe700 1 -- 172.21.15.191:0/1560509248 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f7cfc0103d0 con 0x7f7cf00035c0 2024-03-17T23:48:03.663 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-17T23:48:03.663 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":3,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","modified":"2024-03-17T23:47:58.038625Z","created":"2024-03-17T23:45:34.840429Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-17T23:48:03.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.665+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1560509248 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7ce8039450 msgr2=0x7f7ce803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:03.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.665+0000 7f7d0c63f700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7ce8039450 0x7f7ce803b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009180 tx=0x7f7cf4008040 comp rx=0 tx=0).stop 2024-03-17T23:48:03.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.665+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1560509248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cf00035c0 msgr2=0x7f7d0419f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:03.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.665+0000 7f7d0c63f700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cf00035c0 0x7f7d0419f8b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7d04101ce0 tx=0x7f7cfc005be0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.665+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1560509248 shutdown_connections 2024-03-17T23:48:03.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.665+0000 7f7d0c63f700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7ce8039450 0x7f7ce803b910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.665+0000 7f7d0c63f700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d04199fb0 0x7f7d0419a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.665+0000 7f7d0c63f700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cf00035c0 0x7f7d0419f8b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.665+0000 7f7d0c63f700 1 --2- 172.21.15.191:0/1560509248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d0419fdf0 0x7f7d04199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.665+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1560509248 >> 172.21.15.191:0/1560509248 conn(0x7f7d040fd8c0 msgr2=0x7f7d040fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:03.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.665+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1560509248 shutdown_connections 2024-03-17T23:48:03.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:03.665+0000 7f7d0c63f700 1 -- 172.21.15.191:0/1560509248 wait complete. 2024-03-17T23:48:03.666 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 3 2024-03-17T23:48:04.122 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-17T23:48:04.123 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-17T23:48:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:04 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:04 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:04 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:04 smithi191 ceph-mon[36708]: from='client.? 172.21.15.191:0/1560509248' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-17T23:48:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:04 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:04 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:04.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:04 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:04.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:04 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:04.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:04 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:04.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:04 smithi012 ceph-mon[31541]: from='client.? 172.21.15.191:0/1560509248' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-17T23:48:04.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:04 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:04.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:04 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:04.561 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:48:05.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:05.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: mon.c calling monitor election 2024-03-17T23:48:05.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:48:05.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: mon.a calling monitor election 2024-03-17T23:48:05.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:05.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: monmap epoch 3 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: last_changed 2024-03-17T23:47:58.038625+0000 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: created 2024-03-17T23:45:34.840429+0000 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: min_mon_release 19 (squid) 2024-03-17T23:48:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: election_strategy: 1 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: 2: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.b 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: fsmap 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: osdmap e4: 0 total, 0 up, 0 in 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: mgrmap e13: x(active, since 69s) 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: mon.b (rank 2) addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] is down (out of quorum) 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:05.548 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:05.549 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='client.? 172.21.15.191:0/1560509248' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-17T23:48:05.549 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:05.549 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:06.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: mon.b calling monitor election 2024-03-17T23:48:06.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: mon.b calling monitor election 2024-03-17T23:48:06.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: mon.c calling monitor election 2024-03-17T23:48:06.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: mon.a calling monitor election 2024-03-17T23:48:06.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-17T23:48:06.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: monmap epoch 3 2024-03-17T23:48:06.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:48:06.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: last_changed 2024-03-17T23:47:58.038625+0000 2024-03-17T23:48:06.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: created 2024-03-17T23:45:34.840429+0000 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: min_mon_release 19 (squid) 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: election_strategy: 1 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: 2: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.b 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: fsmap 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: osdmap e4: 0 total, 0 up, 0 in 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: mgrmap e13: x(active, since 71s) 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: Cluster is now healthy 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: overall HEALTH_OK 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: Updating smithi079:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: Updating smithi191:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:48:06.075 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:06.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:06.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:06.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:06.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:05 smithi012 ceph-mon[31541]: Updating smithi012:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:48:06.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: mon.b calling monitor election 2024-03-17T23:48:06.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: mon.b calling monitor election 2024-03-17T23:48:06.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: mon.c calling monitor election 2024-03-17T23:48:06.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: mon.a calling monitor election 2024-03-17T23:48:06.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-17T23:48:06.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: monmap epoch 3 2024-03-17T23:48:06.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:48:06.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: last_changed 2024-03-17T23:47:58.038625+0000 2024-03-17T23:48:06.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: created 2024-03-17T23:45:34.840429+0000 2024-03-17T23:48:06.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: min_mon_release 19 (squid) 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: election_strategy: 1 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: 2: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.b 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: fsmap 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: osdmap e4: 0 total, 0 up, 0 in 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: mgrmap e13: x(active, since 71s) 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: Cluster is now healthy 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: overall HEALTH_OK 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: Updating smithi079:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: Updating smithi191:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:06.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:06.298 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:06.298 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:06.298 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:05 smithi079 ceph-mon[36241]: Updating smithi012:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:48:06.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: mon.b calling monitor election 2024-03-17T23:48:06.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: mon.b calling monitor election 2024-03-17T23:48:06.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: mon.c calling monitor election 2024-03-17T23:48:06.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: mon.a calling monitor election 2024-03-17T23:48:06.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-17T23:48:06.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: monmap epoch 3 2024-03-17T23:48:06.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:48:06.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: last_changed 2024-03-17T23:47:58.038625+0000 2024-03-17T23:48:06.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: created 2024-03-17T23:45:34.840429+0000 2024-03-17T23:48:06.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: min_mon_release 19 (squid) 2024-03-17T23:48:06.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: election_strategy: 1 2024-03-17T23:48:06.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-03-17T23:48:06.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-03-17T23:48:06.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: 2: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.b 2024-03-17T23:48:06.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: fsmap 2024-03-17T23:48:06.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: osdmap e4: 0 total, 0 up, 0 in 2024-03-17T23:48:06.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: mgrmap e13: x(active, since 71s) 2024-03-17T23:48:06.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-17T23:48:06.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: Cluster is now healthy 2024-03-17T23:48:06.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: overall HEALTH_OK 2024-03-17T23:48:06.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: Updating smithi079:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:48:06.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: Updating smithi191:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:48:06.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:06.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:06.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:06.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:06.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:05 smithi191 ceph-mon[36708]: Updating smithi012:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-17T23:48:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.610+0000 7ffa09400700 1 -- 172.21.15.12:0/2463045672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa04102e60 msgr2=0x7ffa04103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.610+0000 7ffa09400700 1 --2- 172.21.15.12:0/2463045672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa04102e60 0x7ffa04103240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400b0a0 tx=0x7ff9f400b3b0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.611+0000 7ffa09400700 1 -- 172.21.15.12:0/2463045672 shutdown_connections 2024-03-17T23:48:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.611+0000 7ffa09400700 1 --2- 172.21.15.12:0/2463045672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa04102e60 0x7ffa04103240 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.611+0000 7ffa09400700 1 -- 172.21.15.12:0/2463045672 >> 172.21.15.12:0/2463045672 conn(0x7ffa04077810 msgr2=0x7ffa040feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.611+0000 7ffa09400700 1 -- 172.21.15.12:0/2463045672 shutdown_connections 2024-03-17T23:48:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.611+0000 7ffa09400700 1 -- 172.21.15.12:0/2463045672 wait complete. 2024-03-17T23:48:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.612+0000 7ffa09400700 1 Processor -- start 2024-03-17T23:48:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.612+0000 7ffa09400700 1 -- start start 2024-03-17T23:48:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.612+0000 7ffa09400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa04102e60 0x7ffa04100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.612+0000 7ffa09400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa04101000 0x7ffa04101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.612+0000 7ffa09400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa041a3c90 0x7ffa041a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.612+0000 7ffa09400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0410c300 con 0x7ffa04101000 2024-03-17T23:48:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.612+0000 7ffa09400700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0410c180 con 0x7ffa041a3c90 2024-03-17T23:48:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.612+0000 7ffa09400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0410c480 con 0x7ffa04102e60 2024-03-17T23:48:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.613+0000 7ffa01e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa04101000 0x7ffa04101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.613+0000 7ffa02800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa04102e60 0x7ffa04100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.613+0000 7ffa01e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa04101000 0x7ffa04101480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36550/0 (socket says 172.21.15.12:36550) 2024-03-17T23:48:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.613+0000 7ffa01e00700 1 -- 172.21.15.12:0/61587179 learned_addr learned my addr 172.21.15.12:0/61587179 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:48:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.613+0000 7ffa03200700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa041a3c90 0x7ffa041a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.613+0000 7ffa01e00700 1 -- 172.21.15.12:0/61587179 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa04102e60 msgr2=0x7ffa04100ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.613+0000 7ffa01e00700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa04102e60 0x7ffa04100ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.613+0000 7ffa01e00700 1 -- 172.21.15.12:0/61587179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa041a3c90 msgr2=0x7ffa041a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.613+0000 7ffa01e00700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa041a3c90 0x7ffa041a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.613+0000 7ffa01e00700 1 -- 172.21.15.12:0/61587179 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa041a67b0 con 0x7ffa04101000 2024-03-17T23:48:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.613+0000 7ffa02800700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa04102e60 0x7ffa04100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.614+0000 7ffa01e00700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa04101000 0x7ffa04101480 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000b780 tx=0x7ff9f000ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.614+0000 7ff9fb400700 1 -- 172.21.15.12:0/61587179 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f0016070 con 0x7ffa04101000 2024-03-17T23:48:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.614+0000 7ff9fb400700 1 -- 172.21.15.12:0/61587179 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9f00036a0 con 0x7ffa04101000 2024-03-17T23:48:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.614+0000 7ffa09400700 1 -- 172.21.15.12:0/61587179 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa041a6aa0 con 0x7ffa04101000 2024-03-17T23:48:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.614+0000 7ff9fb400700 1 -- 172.21.15.12:0/61587179 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f0011af0 con 0x7ffa04101000 2024-03-17T23:48:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.614+0000 7ffa09400700 1 -- 172.21.15.12:0/61587179 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa041a6fe0 con 0x7ffa04101000 2024-03-17T23:48:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.615+0000 7ffa09400700 1 -- 172.21.15.12:0/61587179 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa04069fa0 con 0x7ffa04101000 2024-03-17T23:48:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.615+0000 7ff9fb400700 1 -- 172.21.15.12:0/61587179 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ff9f0019070 con 0x7ffa04101000 2024-03-17T23:48:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.616+0000 7ff9fb400700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff9e8039450 0x7ff9e803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.616+0000 7ff9fb400700 1 -- 172.21.15.12:0/61587179 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff9f004ac60 con 0x7ffa04101000 2024-03-17T23:48:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.616+0000 7ffa02800700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff9e8039450 0x7ff9e803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.617+0000 7ffa02800700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff9e8039450 0x7ff9e803b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400bfd0 tx=0x7ff9f4005040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:06.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.622+0000 7ff9fb400700 1 -- 172.21.15.12:0/61587179 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff9f00199b0 con 0x7ffa04101000 2024-03-17T23:48:06.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.850+0000 7ffa09400700 1 -- 172.21.15.12:0/61587179 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7ffa041a7310 con 0x7ffa04101000 2024-03-17T23:48:06.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.851+0000 7ff9fb400700 1 -- 172.21.15.12:0/61587179 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7ff9f0019c10 con 0x7ffa04101000 2024-03-17T23:48:06.852 INFO:teuthology.orchestra.run.smithi012.stdout:# minimal ceph.conf for 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:48:06.852 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2024-03-17T23:48:06.852 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:48:06.852 INFO:teuthology.orchestra.run.smithi012.stdout: mon_host = [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-17T23:48:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.854+0000 7ffa09400700 1 -- 172.21.15.12:0/61587179 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff9e8039450 msgr2=0x7ff9e803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.854+0000 7ffa09400700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff9e8039450 0x7ff9e803b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400bfd0 tx=0x7ff9f4005040 comp rx=0 tx=0).stop 2024-03-17T23:48:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.854+0000 7ffa09400700 1 -- 172.21.15.12:0/61587179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa04101000 msgr2=0x7ffa04101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.854+0000 7ffa09400700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa04101000 0x7ffa04101480 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000b780 tx=0x7ff9f000ba90 comp rx=0 tx=0).stop 2024-03-17T23:48:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.854+0000 7ffa09400700 1 -- 172.21.15.12:0/61587179 shutdown_connections 2024-03-17T23:48:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.854+0000 7ffa09400700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff9e8039450 0x7ff9e803b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.854+0000 7ffa09400700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa041a3c90 0x7ffa041a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.854+0000 7ffa09400700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa04102e60 0x7ffa04100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.854+0000 7ffa09400700 1 --2- 172.21.15.12:0/61587179 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa04101000 0x7ffa04101480 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.854+0000 7ffa09400700 1 -- 172.21.15.12:0/61587179 >> 172.21.15.12:0/61587179 conn(0x7ffa04077810 msgr2=0x7ffa04103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.855+0000 7ffa09400700 1 -- 172.21.15.12:0/61587179 shutdown_connections 2024-03-17T23:48:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:06.855+0000 7ffa09400700 1 -- 172.21.15.12:0/61587179 wait complete. 2024-03-17T23:48:07.168 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:48:07 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:48:07.039+0000 7fb970000700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-17T23:48:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: Reconfiguring mon.a (unknown last config time)... 2024-03-17T23:48:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-17T23:48:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-17T23:48:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:07.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: Reconfiguring daemon mon.a on smithi012 2024-03-17T23:48:07.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:06 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/61587179' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:07.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:07.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:07.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: Reconfiguring mon.a (unknown last config time)... 2024-03-17T23:48:07.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-17T23:48:07.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-17T23:48:07.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:07.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: Reconfiguring daemon mon.a on smithi012 2024-03-17T23:48:07.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:06 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/61587179' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:07.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:07.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:48:07.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:07.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: Reconfiguring mon.a (unknown last config time)... 2024-03-17T23:48:07.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-17T23:48:07.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-17T23:48:07.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:07.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: Reconfiguring daemon mon.a on smithi012 2024-03-17T23:48:07.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:06 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/61587179' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:07.585 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-17T23:48:07.585 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:48:07.585 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.conf 2024-03-17T23:48:07.667 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:48:07.667 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-17T23:48:07.820 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-17T23:48:07.820 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.conf 2024-03-17T23:48:07.855 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-17T23:48:07.855 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-17T23:48:07.926 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-17T23:48:07.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.conf 2024-03-17T23:48:07.956 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-17T23:48:07.957 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-17T23:48:08.024 INFO:tasks.cephadm:Adding mgr.x on smithi012 2024-03-17T23:48:08.024 INFO:tasks.cephadm:Adding mgr.y on smithi079 2024-03-17T23:48:08.024 INFO:tasks.cephadm:Adding mgr.z on smithi191 2024-03-17T23:48:08.024 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi012=x;smithi079=y;smithi191=z' 2024-03-17T23:48:08.297 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-17T23:48:09.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:09 smithi079 ceph-mon[36241]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:09.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:09 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:09.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:09 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:09.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:09 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-17T23:48:09.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:09 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-17T23:48:09.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:09 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:09.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:09 smithi191 ceph-mon[36708]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:09.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:09 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:09.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:09 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:09.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:09 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-17T23:48:09.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:09 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-17T23:48:09.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:09 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:09.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:09 smithi012 ceph-mon[31541]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:09.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:09.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:09.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-17T23:48:09.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-17T23:48:09.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:09.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.819+0000 7fbd81682700 1 -- 172.21.15.191:0/2581437512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd7c100cf0 msgr2=0x7fbd7c101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:09.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.819+0000 7fbd81682700 1 --2- 172.21.15.191:0/2581437512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd7c100cf0 0x7fbd7c101110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400b0a0 tx=0x7fbd7400b3b0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.819+0000 7fbd81682700 1 -- 172.21.15.191:0/2581437512 shutdown_connections 2024-03-17T23:48:09.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.819+0000 7fbd81682700 1 --2- 172.21.15.191:0/2581437512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd7c100cf0 0x7fbd7c101110 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.819+0000 7fbd81682700 1 -- 172.21.15.191:0/2581437512 >> 172.21.15.191:0/2581437512 conn(0x7fbd7c0fc2b0 msgr2=0x7fbd7c0fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:09.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.819+0000 7fbd81682700 1 -- 172.21.15.191:0/2581437512 shutdown_connections 2024-03-17T23:48:09.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.819+0000 7fbd81682700 1 -- 172.21.15.191:0/2581437512 wait complete. 2024-03-17T23:48:09.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.820+0000 7fbd81682700 1 Processor -- start 2024-03-17T23:48:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.820+0000 7fbd81682700 1 -- start start 2024-03-17T23:48:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.820+0000 7fbd81682700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd7c100cf0 0x7fbd7c197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.820+0000 7fbd81682700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd7c197b60 0x7fbd7c19cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.820+0000 7fbd81682700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbd7c197fe0 0x7fbd7c19f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.820+0000 7fbd81682700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7c106d30 con 0x7fbd7c100cf0 2024-03-17T23:48:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.820+0000 7fbd81682700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7c106bb0 con 0x7fbd7c197fe0 2024-03-17T23:48:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.820+0000 7fbd81682700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7c106eb0 con 0x7fbd7c197b60 2024-03-17T23:48:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.820+0000 7fbd7a7fc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd7c197b60 0x7fbd7c19cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.820+0000 7fbd7a7fc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd7c197b60 0x7fbd7c19cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59200/0 (socket says 172.21.15.191:59200) 2024-03-17T23:48:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.820+0000 7fbd7a7fc700 1 -- 172.21.15.191:0/1485774581 learned_addr learned my addr 172.21.15.191:0/1485774581 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-17T23:48:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.820+0000 7fbd7b7fe700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbd7c197fe0 0x7fbd7c19f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.821+0000 7fbd7a7fc700 1 -- 172.21.15.191:0/1485774581 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbd7c197fe0 msgr2=0x7fbd7c19f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.821+0000 7fbd7a7fc700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbd7c197fe0 0x7fbd7c19f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.821+0000 7fbd7a7fc700 1 -- 172.21.15.191:0/1485774581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd7c100cf0 msgr2=0x7fbd7c197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.821+0000 7fbd7affd700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd7c100cf0 0x7fbd7c197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.821+0000 7fbd7a7fc700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd7c100cf0 0x7fbd7c197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.821+0000 7fbd7a7fc700 1 -- 172.21.15.191:0/1485774581 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd7c19fa30 con 0x7fbd7c197b60 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.821+0000 7fbd7a7fc700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd7c197b60 0x7fbd7c19cbd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400b780 tx=0x7fbd6400bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.821+0000 7fbd73fff700 1 -- 172.21.15.191:0/1485774581 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd64004230 con 0x7fbd7c197b60 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.821+0000 7fbd81682700 1 -- 172.21.15.191:0/1485774581 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd7c19fd20 con 0x7fbd7c197b60 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.821+0000 7fbd73fff700 1 -- 172.21.15.191:0/1485774581 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd640043d0 con 0x7fbd7c197b60 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.821+0000 7fbd73fff700 1 -- 172.21.15.191:0/1485774581 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd64011c60 con 0x7fbd7c197b60 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.821+0000 7fbd81682700 1 -- 172.21.15.191:0/1485774581 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd7c1a0280 con 0x7fbd7c197b60 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.822+0000 7fbd73fff700 1 -- 172.21.15.191:0/1485774581 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fbd6401a400 con 0x7fbd7c197b60 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.822+0000 7fbd81682700 1 -- 172.21.15.191:0/1485774581 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd7c063900 con 0x7fbd7c197b60 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.822+0000 7fbd73fff700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbd68039450 0x7fbd6803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.822+0000 7fbd73fff700 1 -- 172.21.15.191:0/1485774581 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbd6404bf70 con 0x7fbd7c197b60 2024-03-17T23:48:09.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.823+0000 7fbd7affd700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbd68039450 0x7fbd6803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:09.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.823+0000 7fbd7affd700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbd68039450 0x7fbd6803b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd74009e60 tx=0x7fbd7400b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:09.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:09.827+0000 7fbd73fff700 1 -- 172.21.15.191:0/1485774581 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbd64020500 con 0x7fbd7c197b60 2024-03-17T23:48:10.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.031+0000 7fbd81682700 1 -- 172.21.15.191:0/1485774581 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi079=y;smithi191=z", "target": ["mon-mgr", ""]}) -- 0x7fbd7c062710 con 0x7fbd68039450 2024-03-17T23:48:10.055 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.055+0000 7fbd73fff700 1 -- 172.21.15.191:0/1485774581 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fbd7c062710 con 0x7fbd68039450 2024-03-17T23:48:10.055 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mgr update... 2024-03-17T23:48:10.057 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.057+0000 7fbd81682700 1 -- 172.21.15.191:0/1485774581 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbd68039450 msgr2=0x7fbd6803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:10.057 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.057+0000 7fbd81682700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbd68039450 0x7fbd6803b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd74009e60 tx=0x7fbd7400b7b0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.057 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.057+0000 7fbd81682700 1 -- 172.21.15.191:0/1485774581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd7c197b60 msgr2=0x7fbd7c19cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:10.058 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.057+0000 7fbd81682700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd7c197b60 0x7fbd7c19cbd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400b780 tx=0x7fbd6400bb40 comp rx=0 tx=0).stop 2024-03-17T23:48:10.058 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.057+0000 7fbd81682700 1 -- 172.21.15.191:0/1485774581 shutdown_connections 2024-03-17T23:48:10.058 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.057+0000 7fbd81682700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbd68039450 0x7fbd6803b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.058 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.057+0000 7fbd81682700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbd7c197fe0 0x7fbd7c19f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.058 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.057+0000 7fbd81682700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd7c100cf0 0x7fbd7c197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.058 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.057+0000 7fbd81682700 1 --2- 172.21.15.191:0/1485774581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd7c197b60 0x7fbd7c19cbd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.058 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.057+0000 7fbd81682700 1 -- 172.21.15.191:0/1485774581 >> 172.21.15.191:0/1485774581 conn(0x7fbd7c0fc2b0 msgr2=0x7fbd7c06a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:10.058 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.057+0000 7fbd81682700 1 -- 172.21.15.191:0/1485774581 shutdown_connections 2024-03-17T23:48:10.058 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:48:10.057+0000 7fbd81682700 1 -- 172.21.15.191:0/1485774581 wait complete. 2024-03-17T23:48:10.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:10 smithi191 ceph-mon[36708]: Reconfiguring mon.b (monmap changed)... 2024-03-17T23:48:10.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:10 smithi191 ceph-mon[36708]: Reconfiguring daemon mon.b on smithi079 2024-03-17T23:48:10.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:10 smithi012 ceph-mon[31541]: Reconfiguring mon.b (monmap changed)... 2024-03-17T23:48:10.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:10 smithi012 ceph-mon[31541]: Reconfiguring daemon mon.b on smithi079 2024-03-17T23:48:10.522 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:10 smithi079 ceph-mon[36241]: Reconfiguring mon.b (monmap changed)... 2024-03-17T23:48:10.522 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:10 smithi079 ceph-mon[36241]: Reconfiguring daemon mon.b on smithi079 2024-03-17T23:48:10.675 DEBUG:teuthology.orchestra.run.smithi079:mgr.y> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mgr.y.service 2024-03-17T23:48:10.677 DEBUG:teuthology.orchestra.run.smithi191:mgr.z> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mgr.z.service 2024-03-17T23:48:10.680 INFO:tasks.cephadm:Deploying OSDs... 2024-03-17T23:48:10.680 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:48:10.680 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2024-03-17T23:48:10.744 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-17T23:48:10.744 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2024-03-17T23:48:10.802 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-17T23:48:10.803 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:48:10.803 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 633 Links: 1 2024-03-17T23:48:10.803 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:48:10.803 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:48:10.803 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-17 23:47:06.920111573 +0000 2024-03-17T23:48:10.803 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-17 23:34:45.085143359 +0000 2024-03-17T23:48:10.803 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-17 23:34:45.085143359 +0000 2024-03-17T23:48:10.803 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-17 23:34:45.085143359 +0000 2024-03-17T23:48:10.803 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-17T23:48:10.902 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-17T23:48:10.902 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-17T23:48:10.902 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.0003867 s, 1.3 MB/s 2024-03-17T23:48:10.906 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-17T23:48:10.967 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2024-03-17T23:48:11.018 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-17T23:48:11.019 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:48:11.019 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-17T23:48:11.019 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:48:11.019 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:48:11.019 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-17 23:47:07.032110623 +0000 2024-03-17T23:48:11.019 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-17 23:34:45.703138312 +0000 2024-03-17T23:48:11.019 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-17 23:34:45.703138312 +0000 2024-03-17T23:48:11.019 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-17 23:34:45.703138312 +0000 2024-03-17T23:48:11.019 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-17T23:48:11.108 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-17T23:48:11.108 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-17T23:48:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000332623 s, 1.5 MB/s 2024-03-17T23:48:11.112 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-17T23:48:11.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:11 smithi012 ceph-mon[31541]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:11.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:11 smithi012 ceph-mon[31541]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi079=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:48:11.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:11 smithi012 ceph-mon[31541]: Saving service mgr spec with placement smithi012=x;smithi079=y;smithi191=z;count:3 2024-03-17T23:48:11.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:11.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:11.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:11.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:11 smithi012 ceph-mon[31541]: Reconfiguring mon.c (monmap changed)... 2024-03-17T23:48:11.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-17T23:48:11.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-17T23:48:11.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:11.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:11 smithi012 ceph-mon[31541]: Reconfiguring daemon mon.c on smithi191 2024-03-17T23:48:11.175 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2024-03-17T23:48:11.230 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-17T23:48:11.231 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:48:11.231 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-03-17T23:48:11.231 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:48:11.231 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:48:11.231 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-17 23:47:07.102110029 +0000 2024-03-17T23:48:11.231 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-17 23:34:46.311133346 +0000 2024-03-17T23:48:11.231 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-17 23:34:46.311133346 +0000 2024-03-17T23:48:11.231 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-17 23:34:46.311133346 +0000 2024-03-17T23:48:11.231 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-17T23:48:11.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:11 smithi191 ceph-mon[36708]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:11.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:11 smithi191 ceph-mon[36708]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi079=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:48:11.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:11 smithi191 ceph-mon[36708]: Saving service mgr spec with placement smithi012=x;smithi079=y;smithi191=z;count:3 2024-03-17T23:48:11.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:11.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:11.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:11.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:11 smithi191 ceph-mon[36708]: Reconfiguring mon.c (monmap changed)... 2024-03-17T23:48:11.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-17T23:48:11.261 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-17T23:48:11.261 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:11.261 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:11 smithi191 ceph-mon[36708]: Reconfiguring daemon mon.c on smithi191 2024-03-17T23:48:11.319 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-17T23:48:11.320 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-17T23:48:11.320 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000346951 s, 1.5 MB/s 2024-03-17T23:48:11.324 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-17T23:48:11.379 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2024-03-17T23:48:11.429 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-17T23:48:11.429 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:48:11.429 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-03-17T23:48:11.429 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:48:11.429 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:48:11.429 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-17 23:47:07.184109334 +0000 2024-03-17T23:48:11.429 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-17 23:34:46.895128576 +0000 2024-03-17T23:48:11.429 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-17 23:34:46.895128576 +0000 2024-03-17T23:48:11.429 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-17 23:34:46.895128576 +0000 2024-03-17T23:48:11.430 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-17T23:48:11.524 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-17T23:48:11.524 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-17T23:48:11.525 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.00039719 s, 1.3 MB/s 2024-03-17T23:48:11.529 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-17T23:48:11.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:11 smithi079 ceph-mon[36241]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:11.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:11 smithi079 ceph-mon[36241]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi079=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:48:11.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:11 smithi079 ceph-mon[36241]: Saving service mgr spec with placement smithi012=x;smithi079=y;smithi191=z;count:3 2024-03-17T23:48:11.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:11.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:11.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:11.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:11 smithi079 ceph-mon[36241]: Reconfiguring mon.c (monmap changed)... 2024-03-17T23:48:11.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-17T23:48:11.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-17T23:48:11.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:11.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:11 smithi079 ceph-mon[36241]: Reconfiguring daemon mon.c on smithi191 2024-03-17T23:48:11.591 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-17T23:48:11.591 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-03-17T23:48:11.611 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-17T23:48:11.611 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-03-17T23:48:11.669 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-17T23:48:11.669 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:48:11.669 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-17T23:48:11.669 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:48:11.669 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:48:11.669 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-17 23:47:57.513427079 +0000 2024-03-17T23:48:11.669 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-17 23:33:21.609170757 +0000 2024-03-17T23:48:11.669 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-17 23:33:21.609170757 +0000 2024-03-17T23:48:11.670 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-17 23:33:21.609170757 +0000 2024-03-17T23:48:11.670 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-17T23:48:11.739 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-17T23:48:11.739 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-17T23:48:11.739 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000294412 s, 1.7 MB/s 2024-03-17T23:48:11.740 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-17T23:48:11.798 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-03-17T23:48:11.855 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-17T23:48:11.855 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:48:11.855 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-17T23:48:11.855 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:48:11.855 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:48:11.855 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-17 23:47:57.513427079 +0000 2024-03-17T23:48:11.855 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-17 23:33:21.960162321 +0000 2024-03-17T23:48:11.855 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-17 23:33:21.960162321 +0000 2024-03-17T23:48:11.855 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-17 23:33:21.960162321 +0000 2024-03-17T23:48:11.856 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-17T23:48:11.923 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-17T23:48:11.923 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-17T23:48:11.923 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.00022285 s, 2.3 MB/s 2024-03-17T23:48:11.924 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-17T23:48:11.981 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-03-17T23:48:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-17T23:48:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:48:12.037 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-17T23:48:12.037 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:48:12.037 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:48:12.038 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-17 23:47:57.514427055 +0000 2024-03-17T23:48:12.038 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-17 23:33:22.293154318 +0000 2024-03-17T23:48:12.038 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-17 23:33:22.293154318 +0000 2024-03-17T23:48:12.038 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-17 23:33:22.293154318 +0000 2024-03-17T23:48:12.038 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-17T23:48:12.104 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-17T23:48:12.104 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-17T23:48:12.104 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000199106 s, 2.6 MB/s 2024-03-17T23:48:12.105 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-17T23:48:12.162 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-03-17T23:48:12.216 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-17T23:48:12.216 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:48:12.216 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-17T23:48:12.216 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:48:12.216 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:48:12.217 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-17 23:47:57.514427055 +0000 2024-03-17T23:48:12.217 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-17 23:33:22.643145906 +0000 2024-03-17T23:48:12.217 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-17 23:33:22.643145906 +0000 2024-03-17T23:48:12.217 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-17 23:33:22.643145906 +0000 2024-03-17T23:48:12.217 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-17T23:48:12.282 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-17T23:48:12.282 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-17T23:48:12.282 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000209496 s, 2.4 MB/s 2024-03-17T23:48:12.283 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-17T23:48:12.339 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-17T23:48:12.339 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-03-17T23:48:12.354 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-17T23:48:12.354 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-03-17T23:48:12.409 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-17T23:48:12.409 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:48:12.410 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-17T23:48:12.410 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:48:12.410 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:48:12.410 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-17 23:47:47.649458245 +0000 2024-03-17T23:48:12.410 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-17 23:33:27.299763967 +0000 2024-03-17T23:48:12.410 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-17 23:33:27.299763967 +0000 2024-03-17T23:48:12.410 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-17 23:33:27.299763967 +0000 2024-03-17T23:48:12.410 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-17T23:48:12.477 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-17T23:48:12.477 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-17T23:48:12.477 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000274926 s, 1.9 MB/s 2024-03-17T23:48:12.478 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-17T23:48:12.534 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-03-17T23:48:12.588 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-17T23:48:12.588 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:48:12.588 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-17T23:48:12.588 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:48:12.588 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:48:12.588 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-17 23:47:47.650458226 +0000 2024-03-17T23:48:12.588 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-17 23:33:27.654757423 +0000 2024-03-17T23:48:12.589 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-17 23:33:27.654757423 +0000 2024-03-17T23:48:12.589 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-17 23:33:27.654757423 +0000 2024-03-17T23:48:12.589 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-17T23:48:12.654 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-17T23:48:12.654 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-17T23:48:12.654 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000178751 s, 2.9 MB/s 2024-03-17T23:48:12.655 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-17T23:48:12.713 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-03-17T23:48:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-17T23:48:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:48:12.769 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-17T23:48:12.769 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:48:12.769 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:48:12.769 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-17 23:47:47.650458226 +0000 2024-03-17T23:48:12.769 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-17 23:33:27.998751080 +0000 2024-03-17T23:48:12.769 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-17 23:33:27.998751080 +0000 2024-03-17T23:48:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-17 23:33:27.998751080 +0000 2024-03-17T23:48:12.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-17T23:48:12.834 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-17T23:48:12.834 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-17T23:48:12.834 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000214328 s, 2.4 MB/s 2024-03-17T23:48:12.835 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-17T23:48:12.891 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-03-17T23:48:12.947 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-17T23:48:12.947 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:48:12.947 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-17T23:48:12.947 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:48:12.947 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:48:12.947 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-17 23:47:47.650458226 +0000 2024-03-17T23:48:12.947 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-17 23:33:28.336744849 +0000 2024-03-17T23:48:12.947 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-17 23:33:28.336744849 +0000 2024-03-17T23:48:12.947 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-17 23:33:28.336744849 +0000 2024-03-17T23:48:12.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-17T23:48:13.010 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:13.010 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:13.010 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:13.010 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:13.010 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:13.010 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:13.011 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-17T23:48:13.011 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-17T23:48:13.011 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-17T23:48:13.011 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:13.012 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-17T23:48:13.012 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-17T23:48:13.012 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000226801 s, 2.3 MB/s 2024-03-17T23:48:13.013 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-17T23:48:13.069 INFO:tasks.cephadm:Deploying osd.0 on smithi012 with /dev/vg_nvme/lv_4... 2024-03-17T23:48:13.069 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-17T23:48:13.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:13.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:13.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:13.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:13.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:13.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:13.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-17T23:48:13.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-17T23:48:13.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-17T23:48:13.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:13.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:13.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:13.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:13.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:13.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:13.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:13.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-17T23:48:13.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-17T23:48:13.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-17T23:48:13.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:13.528 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:48:13.944 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:13 smithi012 ceph-mon[31541]: Deploying daemon mgr.z on smithi191 2024-03-17T23:48:13.944 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:13 smithi012 ceph-mon[31541]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:14.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:13 smithi191 ceph-mon[36708]: Deploying daemon mgr.z on smithi191 2024-03-17T23:48:14.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:13 smithi191 ceph-mon[36708]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:14.092 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:13 smithi191 systemd[1]: Starting Ceph mgr.z for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-17T23:48:14.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:13 smithi079 ceph-mon[36241]: Deploying daemon mgr.z on smithi191 2024-03-17T23:48:14.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:13 smithi079 ceph-mon[36241]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:14.414 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:14 smithi191 podman[38563]: 2024-03-17 23:48:14.03364485 +0000 UTC m=+0.017879018 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-17T23:48:14.414 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:14 smithi191 podman[38563]: 2024-03-17 23:48:14.14386255 +0000 UTC m=+0.128096737 container create 436c47a81ab9c856238d0210480aa4b1a899b34ec6efd03631320bacae943dfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-17T23:48:14.414 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:14 smithi191 podman[38563]: 2024-03-17 23:48:14.338365421 +0000 UTC m=+0.322599602 container init 436c47a81ab9c856238d0210480aa4b1a899b34ec6efd03631320bacae943dfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-17T23:48:14.414 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:14 smithi191 podman[38563]: 2024-03-17 23:48:14.340897141 +0000 UTC m=+0.325131304 container start 436c47a81ab9c856238d0210480aa4b1a899b34ec6efd03631320bacae943dfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-17T23:48:14.414 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:14 smithi191 ceph-mgr[38582]: -- 172.21.15.191:0/249586534 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f46b1c05a0 con 0x55f46b194800 2024-03-17T23:48:14.414 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:14 smithi191 bash[38563]: 436c47a81ab9c856238d0210480aa4b1a899b34ec6efd03631320bacae943dfc 2024-03-17T23:48:14.841 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:14 smithi191 systemd[1]: Started Ceph mgr.z for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:48:15.341 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:14 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:14.880+0000 7f8a709ca200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-17T23:48:15.341 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:14 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:14.972+0000 7f8a709ca200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-17T23:48:15.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:15 smithi012 ceph-mon[31541]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:15.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:15.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:15.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:15.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-17T23:48:15.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-17T23:48:15.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-17T23:48:15.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:15.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:15 smithi012 ceph-mon[31541]: Deploying daemon mgr.y on smithi079 2024-03-17T23:48:15.763 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:15 smithi079 ceph-mon[36241]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:15.763 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:15.763 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:15.763 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:15.763 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-17T23:48:15.763 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-17T23:48:15.763 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-17T23:48:15.763 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:15.763 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:15 smithi079 ceph-mon[36241]: Deploying daemon mgr.y on smithi079 2024-03-17T23:48:15.841 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:15 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:15.609+0000 7f8a709ca200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-17T23:48:15.841 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:15 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:15.700+0000 7f8a709ca200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-17T23:48:15.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:15 smithi191 ceph-mon[36708]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:15.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:15.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:15.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:15.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-17T23:48:15.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-17T23:48:15.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-17T23:48:15.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:15.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:15 smithi191 ceph-mon[36708]: Deploying daemon mgr.y on smithi079 2024-03-17T23:48:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-17T23:48:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-17T23:48:16.276 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-03-17T23:48:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-03-17T23:48:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0346232 s, 303 MB/s 2024-03-17T23:48:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-03-17T23:48:16.283 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:15 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:15.841+0000 7f8a709ca200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-17T23:48:16.283 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:15 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:15.935+0000 7f8a709ca200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-17T23:48:16.284 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:16.017+0000 7f8a709ca200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-17T23:48:16.284 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:16.282+0000 7f8a709ca200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-17T23:48:16.569 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:16.390+0000 7f8a709ca200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-17T23:48:16.570 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:16.475+0000 7f8a709ca200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-17T23:48:16.570 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:16.568+0000 7f8a709ca200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-17T23:48:16.796 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:16 smithi079 podman[37617]: 2024-03-17 23:48:16.690673299 +0000 UTC m=+0.132972953 container create 48dbac523852c5d1586563fe778d996b0ca9a604fd2aeea7a7e07158d28f0e0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-17T23:48:16.841 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:16.745+0000 7f8a709ca200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-17T23:48:17.296 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:16 smithi079 podman[37617]: 2024-03-17 23:48:16.901228009 +0000 UTC m=+0.343527672 container init 48dbac523852c5d1586563fe778d996b0ca9a604fd2aeea7a7e07158d28f0e0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-17T23:48:17.296 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:16 smithi079 podman[37617]: 2024-03-17 23:48:16.903876113 +0000 UTC m=+0.346175767 container start 48dbac523852c5d1586563fe778d996b0ca9a604fd2aeea7a7e07158d28f0e0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-03-17T23:48:17.296 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:16 smithi079 ceph-mgr[37636]: -- 172.21.15.79:0/197980011 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557ec26b25a0 con 0x557ec2686c00 2024-03-17T23:48:17.296 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:16 smithi079 bash[37617]: 48dbac523852c5d1586563fe778d996b0ca9a604fd2aeea7a7e07158d28f0e0d 2024-03-17T23:48:17.296 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:16 smithi079 systemd[1]: Started Ceph mgr.y for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:48:17.374 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi012:vg_nvme/lv_4 2024-03-17T23:48:17.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:17 smithi012 ceph-mon[31541]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:17.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:17.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:17.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:17.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:17.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:17.796 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:17 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:17.446+0000 7f8745bda200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-17T23:48:17.796 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:17 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:17.543+0000 7f8745bda200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-17T23:48:17.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:17 smithi079 ceph-mon[36241]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:17.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:17.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:17.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:17.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:17.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:17.820 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:48:17.841 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:17 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:17.727+0000 7f8a709ca200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-17T23:48:17.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:17 smithi191 ceph-mon[36708]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:17.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:17.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:17.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:17.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:17.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:18.231 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:17 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:17.869+0000 7f8a709ca200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-17T23:48:18.527 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:18.229+0000 7f8a709ca200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-17T23:48:18.527 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:18.321+0000 7f8a709ca200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-17T23:48:18.545 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:18 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:18.218+0000 7f8745bda200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-17T23:48:18.546 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:18 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:18.314+0000 7f8745bda200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-17T23:48:18.546 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:18 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:18.459+0000 7f8745bda200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-17T23:48:18.841 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:18.526+0000 7f8a709ca200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:48:18.841 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:18.720+0000 7f8a709ca200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-17T23:48:18.911 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:18 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:18.554+0000 7f8745bda200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-17T23:48:18.911 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:18 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:18.645+0000 7f8745bda200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-17T23:48:18.911 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:18 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:18.910+0000 7f8745bda200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-17T23:48:19.195 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:19.018+0000 7f8745bda200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-17T23:48:19.195 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:19.102+0000 7f8745bda200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-17T23:48:19.196 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:19.194+0000 7f8745bda200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-17T23:48:19.266 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:19.005+0000 7f8a709ca200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:48:19.266 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:19.264+0000 7f8a709ca200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-17T23:48:19.545 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:19.368+0000 7f8745bda200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-17T23:48:19.591 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:19.495+0000 7f8a709ca200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-17T23:48:19.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.767+0000 7fbaa5c00700 1 -- 172.21.15.12:0/4091403017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa01056c0 msgr2=0x7fbaa0107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.767+0000 7fbaa5c00700 1 --2- 172.21.15.12:0/4091403017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa01056c0 0x7fbaa0107ab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fba88009990 tx=0x7fba88009ca0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.767+0000 7fbaa5c00700 1 -- 172.21.15.12:0/4091403017 shutdown_connections 2024-03-17T23:48:19.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.767+0000 7fbaa5c00700 1 --2- 172.21.15.12:0/4091403017 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaa0107ff0 0x7fbaa010a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.767+0000 7fbaa5c00700 1 --2- 172.21.15.12:0/4091403017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaa0102d90 0x7fbaa0105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.767+0000 7fbaa5c00700 1 --2- 172.21.15.12:0/4091403017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa01056c0 0x7fbaa0107ab0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.767+0000 7fbaa5c00700 1 -- 172.21.15.12:0/4091403017 >> 172.21.15.12:0/4091403017 conn(0x7fbaa00fc7a0 msgr2=0x7fbaa00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.767+0000 7fbaa5c00700 1 -- 172.21.15.12:0/4091403017 shutdown_connections 2024-03-17T23:48:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.767+0000 7fbaa5c00700 1 -- 172.21.15.12:0/4091403017 wait complete. 2024-03-17T23:48:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.768+0000 7fbaa5c00700 1 Processor -- start 2024-03-17T23:48:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.768+0000 7fbaa5c00700 1 -- start start 2024-03-17T23:48:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.768+0000 7fbaa5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaa0102d90 0x7fbaa019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fbaa5c00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaa01056c0 0x7fbaa019ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fbaa5c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa0107ff0 0x7fbaa01a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fbaa5c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa010df80 con 0x7fbaa0102d90 2024-03-17T23:48:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fbaa5c00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa010de00 con 0x7fbaa01056c0 2024-03-17T23:48:19.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fbaa5c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa010e100 con 0x7fbaa0107ff0 2024-03-17T23:48:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fba97e00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaa01056c0 0x7fbaa019ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fba9ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaa0102d90 0x7fbaa019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fba9f400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa0107ff0 0x7fbaa01a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fba97e00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaa01056c0 0x7fbaa019ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:51886/0 (socket says 172.21.15.12:51886) 2024-03-17T23:48:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fba97e00700 1 -- 172.21.15.12:0/1863145478 learned_addr learned my addr 172.21.15.12:0/1863145478 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:48:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fba9ea00700 1 -- 172.21.15.12:0/1863145478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa0107ff0 msgr2=0x7fbaa01a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fba9ea00700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa0107ff0 0x7fbaa01a1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fba9ea00700 1 -- 172.21.15.12:0/1863145478 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaa01056c0 msgr2=0x7fbaa019ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fba9ea00700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaa01056c0 0x7fbaa019ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fba9ea00700 1 -- 172.21.15.12:0/1863145478 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa01a2380 con 0x7fbaa0102d90 2024-03-17T23:48:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fba97e00700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaa01056c0 0x7fbaa019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.769+0000 7fba9f400700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa0107ff0 0x7fbaa01a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:19.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.770+0000 7fba9ea00700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaa0102d90 0x7fbaa019a720 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fba9000ea20 tx=0x7fba9000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.770+0000 7fba97400700 1 -- 172.21.15.12:0/1863145478 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9000cbd0 con 0x7fbaa0102d90 2024-03-17T23:48:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.770+0000 7fba97400700 1 -- 172.21.15.12:0/1863145478 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba9000ee60 con 0x7fbaa0102d90 2024-03-17T23:48:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.770+0000 7fbaa5c00700 1 -- 172.21.15.12:0/1863145478 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa01a2670 con 0x7fbaa0102d90 2024-03-17T23:48:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.770+0000 7fba97400700 1 -- 172.21.15.12:0/1863145478 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba90018680 con 0x7fbaa0102d90 2024-03-17T23:48:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.770+0000 7fbaa5c00700 1 -- 172.21.15.12:0/1863145478 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa01a2c00 con 0x7fbaa0102d90 2024-03-17T23:48:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.771+0000 7fbaa5c00700 1 -- 172.21.15.12:0/1863145478 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa0068f90 con 0x7fbaa0102d90 2024-03-17T23:48:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.771+0000 7fba97400700 1 -- 172.21.15.12:0/1863145478 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fba90018820 con 0x7fbaa0102d90 2024-03-17T23:48:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.772+0000 7fba97400700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fba84039200 0x7fba8403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.772+0000 7fba97400700 1 -- 172.21.15.12:0/1863145478 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fba9004c2a0 con 0x7fbaa0102d90 2024-03-17T23:48:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.772+0000 7fba97e00700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fba84039200 0x7fba8403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.772+0000 7fba97e00700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fba84039200 0x7fba8403b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fba8800b2e0 tx=0x7fba8800b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:19.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:19.777+0000 7fba97400700 1 -- 172.21.15.12:0/1863145478 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fba90016330 con 0x7fbaa0102d90 2024-03-17T23:48:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:19 smithi012 ceph-mon[31541]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:19 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:19 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:19.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:19 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:19.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:19 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:19.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:19 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:19.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:19 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:19.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:19 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:19.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:19 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-17T23:48:19.921 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:19 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-17T23:48:19.921 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:19 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:19.921 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:19 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:20.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:20.028+0000 7fbaa5c00700 1 -- 172.21.15.12:0/1863145478 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fbaa0062710 con 0x7fba84039200 2024-03-17T23:48:20.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-mon[36241]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-17T23:48:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-17T23:48:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:19 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:20.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-mon[36708]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:20.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:20.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:20.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:20.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:20.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:20.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:20.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:20.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-17T23:48:20.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-17T23:48:20.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:20.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:19 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:20.591 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:20.284+0000 7f8a709ca200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-17T23:48:20.686 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:20 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:20.324+0000 7f8745bda200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-17T23:48:20.686 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:20 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:20.453+0000 7f8745bda200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-17T23:48:20.946 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:20.849+0000 7f8a709ca200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-17T23:48:20.946 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:20.944+0000 7f8a709ca200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-17T23:48:20.946 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:20 smithi191 ceph-mon[36708]: Reconfiguring mgr.x (unknown last config time)... 2024-03-17T23:48:20.946 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:20 smithi191 ceph-mon[36708]: Reconfiguring daemon mgr.x on smithi012 2024-03-17T23:48:20.946 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:20 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:48:20.946 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:20 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:48:20.946 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:20 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:21.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:20 smithi079 ceph-mon[36241]: Reconfiguring mgr.x (unknown last config time)... 2024-03-17T23:48:21.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:20 smithi079 ceph-mon[36241]: Reconfiguring daemon mgr.x on smithi012 2024-03-17T23:48:21.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:20 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:48:21.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:20 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:48:21.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:20 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:21.046 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:20 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:20.782+0000 7f8745bda200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-17T23:48:21.046 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:20 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:20.870+0000 7f8745bda200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-17T23:48:21.103 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:20 smithi012 ceph-mon[31541]: Reconfiguring mgr.x (unknown last config time)... 2024-03-17T23:48:21.103 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:20 smithi012 ceph-mon[31541]: Reconfiguring daemon mgr.x on smithi012 2024-03-17T23:48:21.103 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:20 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:48:21.103 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:20 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:48:21.103 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:20 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:21.341 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z[38578]: 2024-03-17T23:48:21.156+0000 7f8a709ca200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-17T23:48:21.521 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:21.062+0000 7f8745bda200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:48:21.521 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:21.242+0000 7f8745bda200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-17T23:48:21.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:21.709+0000 7fba97400700 1 -- 172.21.15.12:0/1863145478 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 14) ==== 91698+0+0 (secure 0 0 0) 0x7fba90010430 con 0x7fbaa0102d90 2024-03-17T23:48:21.780 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:21.520+0000 7f8745bda200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:48:21.780 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-mon[36241]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:21.781 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-mon[36241]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:48:21.781 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-mon[36241]: from='mgr.? 172.21.15.191:0/1370446145' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-17T23:48:21.781 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-mon[36241]: Standby manager daemon z started 2024-03-17T23:48:21.781 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-mon[36241]: from='mgr.? 172.21.15.191:0/1370446145' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-17T23:48:21.781 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-mon[36241]: from='mgr.? 172.21.15.191:0/1370446145' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-17T23:48:21.781 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-mon[36241]: from='mgr.? 172.21.15.191:0/1370446145' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-17T23:48:21.781 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:21.781 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:21.781 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:21.781 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:21.781 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:21.781 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:21.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:21 smithi012 ceph-mon[31541]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:21.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:21 smithi012 ceph-mon[31541]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:48:21.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:21 smithi012 ceph-mon[31541]: from='mgr.? 172.21.15.191:0/1370446145' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-17T23:48:21.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:21 smithi012 ceph-mon[31541]: Standby manager daemon z started 2024-03-17T23:48:21.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:21 smithi012 ceph-mon[31541]: from='mgr.? 172.21.15.191:0/1370446145' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-17T23:48:21.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:21 smithi012 ceph-mon[31541]: from='mgr.? 172.21.15.191:0/1370446145' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-17T23:48:21.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:21 smithi012 ceph-mon[31541]: from='mgr.? 172.21.15.191:0/1370446145' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-17T23:48:21.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:21 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:21.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:21 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:21.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:21 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:21.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:21 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:21.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:21 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:21.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:21 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:22.046 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:21 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:21.779+0000 7f8745bda200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-17T23:48:22.046 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:22 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:22.012+0000 7f8745bda200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-17T23:48:22.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-mon[36708]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:22.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-mon[36708]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:48:22.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-mon[36708]: from='mgr.? 172.21.15.191:0/1370446145' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-17T23:48:22.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-mon[36708]: Standby manager daemon z started 2024-03-17T23:48:22.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-mon[36708]: from='mgr.? 172.21.15.191:0/1370446145' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-17T23:48:22.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-mon[36708]: from='mgr.? 172.21.15.191:0/1370446145' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-17T23:48:22.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-mon[36708]: from='mgr.? 172.21.15.191:0/1370446145' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-17T23:48:22.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:22.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:22.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:22.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:22.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:22.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:21 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:22.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:22 smithi012 ceph-mon[31541]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-17T23:48:22.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:22 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-17T23:48:23.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:22 smithi079 ceph-mon[36241]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-17T23:48:23.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:22 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-17T23:48:23.046 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:22 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:22.794+0000 7f8745bda200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-17T23:48:23.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:22 smithi191 ceph-mon[36708]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-17T23:48:23.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:22 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-17T23:48:23.667 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:23 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:23.356+0000 7f8745bda200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-17T23:48:23.667 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:23 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:23.452+0000 7f8745bda200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-17T23:48:23.667 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 17 23:48:23 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y[37632]: 2024-03-17T23:48:23.666+0000 7f8745bda200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-17T23:48:23.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:23.739+0000 7fba97400700 1 -- 172.21.15.12:0/1863145478 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7fba90018b10 con 0x7fbaa0102d90 2024-03-17T23:48:24.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:23 smithi079 ceph-mon[36241]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:24.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:23 smithi079 ceph-mon[36241]: Standby manager daemon y started 2024-03-17T23:48:24.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:23 smithi079 ceph-mon[36241]: from='mgr.? 172.21.15.79:0/138640591' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-17T23:48:24.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:23 smithi079 ceph-mon[36241]: from='mgr.? 172.21.15.79:0/138640591' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-17T23:48:24.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:23 smithi079 ceph-mon[36241]: from='mgr.? 172.21.15.79:0/138640591' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-17T23:48:24.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:23 smithi079 ceph-mon[36241]: from='mgr.? 172.21.15.79:0/138640591' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-17T23:48:24.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:23 smithi191 ceph-mon[36708]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:24.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:23 smithi191 ceph-mon[36708]: Standby manager daemon y started 2024-03-17T23:48:24.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:23 smithi191 ceph-mon[36708]: from='mgr.? 172.21.15.79:0/138640591' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-17T23:48:24.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:23 smithi191 ceph-mon[36708]: from='mgr.? 172.21.15.79:0/138640591' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-17T23:48:24.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:23 smithi191 ceph-mon[36708]: from='mgr.? 172.21.15.79:0/138640591' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-17T23:48:24.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:23 smithi191 ceph-mon[36708]: from='mgr.? 172.21.15.79:0/138640591' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-17T23:48:24.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:23 smithi012 ceph-mon[31541]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:24.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:23 smithi012 ceph-mon[31541]: Standby manager daemon y started 2024-03-17T23:48:24.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:23 smithi012 ceph-mon[31541]: from='mgr.? 172.21.15.79:0/138640591' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-17T23:48:24.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:23 smithi012 ceph-mon[31541]: from='mgr.? 172.21.15.79:0/138640591' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-17T23:48:24.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:23 smithi012 ceph-mon[31541]: from='mgr.? 172.21.15.79:0/138640591' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-17T23:48:24.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:23 smithi012 ceph-mon[31541]: from='mgr.? 172.21.15.79:0/138640591' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-17T23:48:25.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:24 smithi079 ceph-mon[36241]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-03-17T23:48:25.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:24 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-17T23:48:25.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:24 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3299865113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b91e248b-f111-4dc6-9dc8-07fd0195699a"}]: dispatch 2024-03-17T23:48:25.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:24 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b91e248b-f111-4dc6-9dc8-07fd0195699a"}]: dispatch 2024-03-17T23:48:25.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:24 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b91e248b-f111-4dc6-9dc8-07fd0195699a"}]': finished 2024-03-17T23:48:25.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:24 smithi079 ceph-mon[36241]: osdmap e5: 1 total, 0 up, 1 in 2024-03-17T23:48:25.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:24 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:25.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:24 smithi191 ceph-mon[36708]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-03-17T23:48:25.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:24 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-17T23:48:25.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:24 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3299865113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b91e248b-f111-4dc6-9dc8-07fd0195699a"}]: dispatch 2024-03-17T23:48:25.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:24 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b91e248b-f111-4dc6-9dc8-07fd0195699a"}]: dispatch 2024-03-17T23:48:25.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:24 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b91e248b-f111-4dc6-9dc8-07fd0195699a"}]': finished 2024-03-17T23:48:25.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:24 smithi191 ceph-mon[36708]: osdmap e5: 1 total, 0 up, 1 in 2024-03-17T23:48:25.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:24 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:25.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:24 smithi012 ceph-mon[31541]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-03-17T23:48:25.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:24 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-17T23:48:25.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:24 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3299865113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b91e248b-f111-4dc6-9dc8-07fd0195699a"}]: dispatch 2024-03-17T23:48:25.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:24 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b91e248b-f111-4dc6-9dc8-07fd0195699a"}]: dispatch 2024-03-17T23:48:25.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:24 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b91e248b-f111-4dc6-9dc8-07fd0195699a"}]': finished 2024-03-17T23:48:25.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:24 smithi012 ceph-mon[31541]: osdmap e5: 1 total, 0 up, 1 in 2024-03-17T23:48:25.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:24 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:25.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:25 smithi191 ceph-mon[36708]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:25.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:25 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/4138073345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:48:26.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:25 smithi079 ceph-mon[36241]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:26.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:25 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/4138073345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:48:26.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:25 smithi012 ceph-mon[31541]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:26.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:25 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4138073345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:48:28.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:27 smithi079 ceph-mon[36241]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:28.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:27 smithi191 ceph-mon[36708]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:28.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:27 smithi012 ceph-mon[31541]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:29.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:29 smithi012 ceph-mon[31541]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:30.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:29 smithi079 ceph-mon[36241]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:30.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:29 smithi191 ceph-mon[36708]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:31.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:31 smithi012 ceph-mon[31541]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:32.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:31 smithi079 ceph-mon[36241]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:32.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:31 smithi191 ceph-mon[36708]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:33.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:33 smithi012 ceph-mon[31541]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:33.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-17T23:48:33.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:34.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:33 smithi079 ceph-mon[36241]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:34.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-17T23:48:34.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:34.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:33 smithi191 ceph-mon[36708]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:34.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-17T23:48:34.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:34.944 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:34 smithi012 ceph-mon[31541]: Deploying daemon osd.0 on smithi012 2024-03-17T23:48:35.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:34 smithi079 ceph-mon[36241]: Deploying daemon osd.0 on smithi012 2024-03-17T23:48:35.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:34 smithi191 ceph-mon[36708]: Deploying daemon osd.0 on smithi012 2024-03-17T23:48:36.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:35 smithi079 ceph-mon[36241]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:36.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:35 smithi191 ceph-mon[36708]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:36.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:35 smithi012 ceph-mon[31541]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:37.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:37 smithi012 ceph-mon[31541]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:38.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:37 smithi079 ceph-mon[36241]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:38.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:37 smithi191 ceph-mon[36708]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:40.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:39 smithi079 ceph-mon[36241]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:40.078 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:39 smithi191 ceph-mon[36708]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:40.167 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:39 smithi012 ceph-mon[31541]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:42.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:41 smithi079 ceph-mon[36241]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:42.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:41 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:42.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:41 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:42.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:41 smithi191 ceph-mon[36708]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:42.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:41 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:42.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:41 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:42.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:41 smithi012 ceph-mon[31541]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:42.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:41 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:42.094 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:41 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:42.962 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:42 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:43.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:42 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:43.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:42 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:44.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:43 smithi079 ceph-mon[36241]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:44.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:43 smithi191 ceph-mon[36708]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:44.116 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:43 smithi012 ceph-mon[31541]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:45.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:45 smithi012 ceph-mon[31541]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:45.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:45 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:45.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:45 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:45.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:45 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:45.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:45 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:45.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:45 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:45.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:45 smithi012 ceph-mon[31541]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-17T23:48:46.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:45 smithi079 ceph-mon[36241]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:46.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:45 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:46.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:45 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:46.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:45 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:46.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:45 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:46.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:45 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:46.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:45 smithi079 ceph-mon[36241]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-17T23:48:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:45 smithi191 ceph-mon[36708]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:45 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:45 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:45 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:45 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:45 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:45 smithi191 ceph-mon[36708]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-17T23:48:46.481 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 0 on host 'smithi012' 2024-03-17T23:48:46.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:46.478+0000 7fba97400700 1 -- 172.21.15.12:0/1863145478 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbaa0062710 con 0x7fba84039200 2024-03-17T23:48:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:46.482+0000 7fbaa5c00700 1 -- 172.21.15.12:0/1863145478 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fba84039200 msgr2=0x7fba8403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:46.482+0000 7fbaa5c00700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fba84039200 0x7fba8403b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fba8800b2e0 tx=0x7fba8800b880 comp rx=0 tx=0).stop 2024-03-17T23:48:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:46.482+0000 7fbaa5c00700 1 -- 172.21.15.12:0/1863145478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaa0102d90 msgr2=0x7fbaa019a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:46.482+0000 7fbaa5c00700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaa0102d90 0x7fbaa019a720 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fba9000ea20 tx=0x7fba9000ed30 comp rx=0 tx=0).stop 2024-03-17T23:48:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:46.482+0000 7fbaa5c00700 1 -- 172.21.15.12:0/1863145478 shutdown_connections 2024-03-17T23:48:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:46.482+0000 7fbaa5c00700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fba84039200 0x7fba8403b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:46.482+0000 7fbaa5c00700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa0107ff0 0x7fbaa01a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:46.482+0000 7fbaa5c00700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbaa0102d90 0x7fbaa019a720 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:46.482+0000 7fbaa5c00700 1 --2- 172.21.15.12:0/1863145478 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaa01056c0 0x7fbaa019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:46.482+0000 7fbaa5c00700 1 -- 172.21.15.12:0/1863145478 >> 172.21.15.12:0/1863145478 conn(0x7fbaa00fc7a0 msgr2=0x7fbaa0102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:46.483+0000 7fbaa5c00700 1 -- 172.21.15.12:0/1863145478 shutdown_connections 2024-03-17T23:48:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:48:46.483+0000 7fbaa5c00700 1 -- 172.21.15.12:0/1863145478 wait complete. 2024-03-17T23:48:47.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:47 smithi079 ceph-mon[36241]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:47.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:47 smithi079 ceph-mon[36241]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-17T23:48:47.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:47 smithi079 ceph-mon[36241]: osdmap e6: 1 total, 0 up, 1 in 2024-03-17T23:48:47.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:47 smithi079 ceph-mon[36241]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:48:47.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:47 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:47.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:47 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:47.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:47 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:47.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:47 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:47.309 DEBUG:teuthology.orchestra.run.smithi012:osd.0> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.0.service 2024-03-17T23:48:47.315 INFO:tasks.cephadm:Deploying osd.1 on smithi079 with /dev/vg_nvme/lv_4... 2024-03-17T23:48:47.316 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-17T23:48:47.549 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.b/config 2024-03-17T23:48:47.576 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:47 smithi012 ceph-mon[31541]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:47.576 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:47 smithi012 ceph-mon[31541]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-17T23:48:47.576 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:47 smithi012 ceph-mon[31541]: osdmap e6: 1 total, 0 up, 1 in 2024-03-17T23:48:47.576 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:47 smithi012 ceph-mon[31541]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:48:47.576 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:47 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:47.576 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:47 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:47.576 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:47 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:47.576 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:47 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:47 smithi191 ceph-mon[36708]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:47 smithi191 ceph-mon[36708]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-17T23:48:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:47 smithi191 ceph-mon[36708]: osdmap e6: 1 total, 0 up, 1 in 2024-03-17T23:48:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:47 smithi191 ceph-mon[36708]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:48:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:47 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:47 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:47.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:47 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:47.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:47 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:48:48.215 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 17 23:48:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0[43943]: 2024-03-17T23:48:47.920+0000 7f5fa2e00700 -1 osd.0 0 waiting for initial osdmap 2024-03-17T23:48:48.216 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 17 23:48:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0[43943]: 2024-03-17T23:48:47.935+0000 7f5f98200700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:48:48.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:48 smithi079 ceph-mon[36241]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-17T23:48:48.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:48 smithi079 ceph-mon[36241]: osdmap e7: 1 total, 0 up, 1 in 2024-03-17T23:48:48.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:48.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:48.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:48 smithi079 ceph-mon[36241]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' 2024-03-17T23:48:48.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:48 smithi191 ceph-mon[36708]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-17T23:48:48.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:48 smithi191 ceph-mon[36708]: osdmap e7: 1 total, 0 up, 1 in 2024-03-17T23:48:48.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:48.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:48.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:48 smithi191 ceph-mon[36708]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' 2024-03-17T23:48:48.639 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:48 smithi012 ceph-mon[31541]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-17T23:48:48.639 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:48 smithi012 ceph-mon[31541]: osdmap e7: 1 total, 0 up, 1 in 2024-03-17T23:48:48.639 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:48.639 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:48.639 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:48 smithi012 ceph-mon[31541]: from='osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279]' entity='osd.0' 2024-03-17T23:48:49.350 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:49 smithi012 ceph-mon[31541]: purged_snaps scrub starts 2024-03-17T23:48:49.351 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:49 smithi012 ceph-mon[31541]: purged_snaps scrub ok 2024-03-17T23:48:49.351 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:49 smithi012 ceph-mon[31541]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:49.351 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:49 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:49.351 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:49 smithi012 ceph-mon[31541]: osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279] boot 2024-03-17T23:48:49.351 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:49 smithi012 ceph-mon[31541]: osdmap e8: 1 total, 1 up, 1 in 2024-03-17T23:48:49.351 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:49 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:49.403 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-17T23:48:49.403 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-17T23:48:49.403 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-03-17T23:48:49.403 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-03-17T23:48:49.403 INFO:teuthology.orchestra.run.smithi079.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0119853 s, 875 MB/s 2024-03-17T23:48:49.403 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-03-17T23:48:49.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:49 smithi079 ceph-mon[36241]: purged_snaps scrub starts 2024-03-17T23:48:49.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:49 smithi079 ceph-mon[36241]: purged_snaps scrub ok 2024-03-17T23:48:49.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:49 smithi079 ceph-mon[36241]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:49.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:49 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:49.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:49 smithi079 ceph-mon[36241]: osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279] boot 2024-03-17T23:48:49.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:49 smithi079 ceph-mon[36241]: osdmap e8: 1 total, 1 up, 1 in 2024-03-17T23:48:49.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:49 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:49.585 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:49 smithi191 ceph-mon[36708]: purged_snaps scrub starts 2024-03-17T23:48:49.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:49 smithi191 ceph-mon[36708]: purged_snaps scrub ok 2024-03-17T23:48:49.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:49 smithi191 ceph-mon[36708]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-17T23:48:49.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:49 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:49.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:49 smithi191 ceph-mon[36708]: osd.0 [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279] boot 2024-03-17T23:48:49.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:49 smithi191 ceph-mon[36708]: osdmap e8: 1 total, 1 up, 1 in 2024-03-17T23:48:49.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:49 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-17T23:48:50.025 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi079:vg_nvme/lv_4 2024-03-17T23:48:50.262 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.b/config 2024-03-17T23:48:51.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:51 smithi079 ceph-mon[36241]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:51.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:51 smithi079 ceph-mon[36241]: osdmap e9: 1 total, 1 up, 1 in 2024-03-17T23:48:51.495 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:51 smithi012 ceph-mon[31541]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:51.496 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:51 smithi012 ceph-mon[31541]: osdmap e9: 1 total, 1 up, 1 in 2024-03-17T23:48:51.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:51 smithi191 ceph-mon[36708]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:51.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:51 smithi191 ceph-mon[36708]: osdmap e9: 1 total, 1 up, 1 in 2024-03-17T23:48:51.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.841+0000 7f481ee95700 1 -- 172.21.15.79:0/603182876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4818108090 msgr2=0x7f481810a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.841+0000 7f481ee95700 1 --2- 172.21.15.79:0/603182876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4818108090 0x7f481810a480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f481000b0a0 tx=0x7f481000b3b0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.841+0000 7f481ee95700 1 -- 172.21.15.79:0/603182876 shutdown_connections 2024-03-17T23:48:51.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.841+0000 7f481ee95700 1 --2- 172.21.15.79:0/603182876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4818108090 0x7f481810a480 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.841+0000 7f481ee95700 1 --2- 172.21.15.79:0/603182876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4818102e30 0x7f4818105220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.841+0000 7f481ee95700 1 --2- 172.21.15.79:0/603182876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4818105760 0x7f4818107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.841+0000 7f481ee95700 1 -- 172.21.15.79:0/603182876 >> 172.21.15.79:0/603182876 conn(0x7f48180fc7c0 msgr2=0x7f48180febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:51.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.841+0000 7f481ee95700 1 -- 172.21.15.79:0/603182876 shutdown_connections 2024-03-17T23:48:51.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.841+0000 7f481ee95700 1 -- 172.21.15.79:0/603182876 wait complete. 2024-03-17T23:48:51.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.841+0000 7f481ee95700 1 Processor -- start 2024-03-17T23:48:51.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.841+0000 7f481ee95700 1 -- start start 2024-03-17T23:48:51.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f481ee95700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4818102e30 0x7f4818198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f481ee95700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4818105760 0x7f4818198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f481ee95700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4818108090 0x7f481819fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f481ee95700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f481810dfb0 con 0x7f4818108090 2024-03-17T23:48:51.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f481ee95700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f481810de30 con 0x7f4818105760 2024-03-17T23:48:51.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f481ee95700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f481810e130 con 0x7f4818102e30 2024-03-17T23:48:51.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f480ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4818105760 0x7f4818198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f480ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4818105760 0x7f4818198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54268/0 (socket says 172.21.15.79:54268) 2024-03-17T23:48:51.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f480ffff700 1 -- 172.21.15.79:0/3402645257 learned_addr learned my addr 172.21.15.79:0/3402645257 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-03-17T23:48:51.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f481cc31700 1 --2- 172.21.15.79:0/3402645257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4818102e30 0x7f4818198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f480ffff700 1 -- 172.21.15.79:0/3402645257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4818102e30 msgr2=0x7f4818198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f480ffff700 1 --2- 172.21.15.79:0/3402645257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4818102e30 0x7f4818198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f480ffff700 1 -- 172.21.15.79:0/3402645257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4818108090 msgr2=0x7f481819fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f480ffff700 1 --2- 172.21.15.79:0/3402645257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4818108090 0x7f481819fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f480ffff700 1 -- 172.21.15.79:0/3402645257 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48181a01e0 con 0x7f4818105760 2024-03-17T23:48:51.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f480ffff700 1 --2- 172.21.15.79:0/3402645257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4818105760 0x7f4818198ac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f480800b530 tx=0x7f480800b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:51.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f480dffb700 1 -- 172.21.15.79:0/3402645257 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808015020 con 0x7f4818105760 2024-03-17T23:48:51.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f480dffb700 1 -- 172.21.15.79:0/3402645257 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4808011d10 con 0x7f4818105760 2024-03-17T23:48:51.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f480dffb700 1 -- 172.21.15.79:0/3402645257 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480800f2e0 con 0x7f4818105760 2024-03-17T23:48:51.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.842+0000 7f481ee95700 1 -- 172.21.15.79:0/3402645257 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48181a04d0 con 0x7f4818105760 2024-03-17T23:48:51.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.843+0000 7f481ee95700 1 -- 172.21.15.79:0/3402645257 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48181a0a40 con 0x7f4818105760 2024-03-17T23:48:51.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.844+0000 7f481ee95700 1 -- 172.21.15.79:0/3402645257 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4818068f90 con 0x7f4818105760 2024-03-17T23:48:51.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.844+0000 7f480dffb700 1 -- 172.21.15.79:0/3402645257 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f4808018020 con 0x7f4818105760 2024-03-17T23:48:51.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.845+0000 7f480dffb700 1 --2- 172.21.15.79:0/3402645257 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f48000a2800 0x7f48000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.845+0000 7f480dffb700 1 -- 172.21.15.79:0/3402645257 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f48080cc810 con 0x7f4818105760 2024-03-17T23:48:51.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.846+0000 7f481cc31700 1 --2- 172.21.15.79:0/3402645257 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f48000a2800 0x7f48000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.847+0000 7f481cc31700 1 --2- 172.21.15.79:0/3402645257 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f48000a2800 0x7f48000a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4814006fd0 tx=0x7f4814008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:51.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:51.850+0000 7f480dffb700 1 -- 172.21.15.79:0/3402645257 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f480801f020 con 0x7f4818105760 2024-03-17T23:48:52.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:48:52.054+0000 7f481ee95700 1 -- 172.21.15.79:0/3402645257 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4818062710 con 0x7f48000a2800 2024-03-17T23:48:52.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:52 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:48:52.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:52 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:48:52.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:52 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:52.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:52 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:48:52.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:52 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:48:52.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:52 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:52.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:52 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:48:52.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:52 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:48:52.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:52 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:53.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:53 smithi012 ceph-mon[31541]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:53.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:53 smithi012 ceph-mon[31541]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:48:53.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:53 smithi012 ceph-mon[31541]: Detected new or changed devices on smithi012 2024-03-17T23:48:53.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:53 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:53.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:53 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:53.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:53 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:48:53.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:53 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi012 to 17095M 2024-03-17T23:48:53.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:53 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:53.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:53 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:53.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:53 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:53.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:53 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:53.717 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:53 smithi079 ceph-mon[36241]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:53.717 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:53 smithi079 ceph-mon[36241]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:48:53.717 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:53 smithi079 ceph-mon[36241]: Detected new or changed devices on smithi012 2024-03-17T23:48:53.717 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:53 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:53.717 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:53 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:53.717 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:53 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:48:53.717 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:53 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi012 to 17095M 2024-03-17T23:48:53.717 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:53 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:53.717 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:53 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:53.717 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:53 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:53.718 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:53 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:53 smithi191 ceph-mon[36708]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:53 smithi191 ceph-mon[36708]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:48:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:53 smithi191 ceph-mon[36708]: Detected new or changed devices on smithi012 2024-03-17T23:48:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:53 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:53.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:53 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:53.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:53 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:48:53.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:53 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi012 to 17095M 2024-03-17T23:48:53.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:53 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:53.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:53 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:48:53.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:53 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:48:53.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:53 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:48:55.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:55 smithi012 ceph-mon[31541]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:55.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:55 smithi012 ceph-mon[31541]: from='client.? 172.21.15.79:0/789946037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13bf3ff0-c8b3-4c24-9960-c6568256772f"}]: dispatch 2024-03-17T23:48:55.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:55 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13bf3ff0-c8b3-4c24-9960-c6568256772f"}]: dispatch 2024-03-17T23:48:55.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:55 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13bf3ff0-c8b3-4c24-9960-c6568256772f"}]': finished 2024-03-17T23:48:55.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:55 smithi012 ceph-mon[31541]: osdmap e10: 2 total, 1 up, 2 in 2024-03-17T23:48:55.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:48:55.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:55 smithi012 ceph-mon[31541]: from='client.? 172.21.15.79:0/4291354879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:48:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:55 smithi079 ceph-mon[36241]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:55 smithi079 ceph-mon[36241]: from='client.? 172.21.15.79:0/789946037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13bf3ff0-c8b3-4c24-9960-c6568256772f"}]: dispatch 2024-03-17T23:48:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:55 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13bf3ff0-c8b3-4c24-9960-c6568256772f"}]: dispatch 2024-03-17T23:48:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:55 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13bf3ff0-c8b3-4c24-9960-c6568256772f"}]': finished 2024-03-17T23:48:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:55 smithi079 ceph-mon[36241]: osdmap e10: 2 total, 1 up, 2 in 2024-03-17T23:48:55.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:48:55.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:55 smithi079 ceph-mon[36241]: from='client.? 172.21.15.79:0/4291354879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:48:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:55 smithi191 ceph-mon[36708]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:55 smithi191 ceph-mon[36708]: from='client.? 172.21.15.79:0/789946037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13bf3ff0-c8b3-4c24-9960-c6568256772f"}]: dispatch 2024-03-17T23:48:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:55 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13bf3ff0-c8b3-4c24-9960-c6568256772f"}]: dispatch 2024-03-17T23:48:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:55 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13bf3ff0-c8b3-4c24-9960-c6568256772f"}]': finished 2024-03-17T23:48:55.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:55 smithi191 ceph-mon[36708]: osdmap e10: 2 total, 1 up, 2 in 2024-03-17T23:48:55.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:48:55.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:55 smithi191 ceph-mon[36708]: from='client.? 172.21.15.79:0/4291354879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:48:57.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:57 smithi012 ceph-mon[31541]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:57.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:57 smithi079 ceph-mon[36241]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:57 smithi191 ceph-mon[36708]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:59.653 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:48:59 smithi079 ceph-mon[36241]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:59.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:48:59 smithi012 ceph-mon[31541]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:48:59.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:48:59 smithi191 ceph-mon[36708]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:01.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:01 smithi079 ceph-mon[36241]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:01.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:01 smithi012 ceph-mon[31541]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:01.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:01 smithi191 ceph-mon[36708]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:02.626 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:02 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-17T23:49:02.626 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:02 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:02.626 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:02 smithi079 ceph-mon[36241]: Deploying daemon osd.1 on smithi079 2024-03-17T23:49:02.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:02 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-17T23:49:02.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:02 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:02.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:02 smithi012 ceph-mon[31541]: Deploying daemon osd.1 on smithi079 2024-03-17T23:49:02.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:02 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-17T23:49:02.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:02 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:02.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:02 smithi191 ceph-mon[36708]: Deploying daemon osd.1 on smithi079 2024-03-17T23:49:03.641 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:03 smithi079 ceph-mon[36241]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:03.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:03 smithi012 ceph-mon[31541]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:03.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:03 smithi191 ceph-mon[36708]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:05.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:05 smithi079 ceph-mon[36241]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:05.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:05 smithi012 ceph-mon[31541]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:05.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:05 smithi191 ceph-mon[36708]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:07.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:07 smithi012 ceph-mon[31541]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:07.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:07 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:07.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:07 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:07.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:07 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:49:07.778 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:07 smithi079 ceph-mon[36241]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:07.778 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:07 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:07.778 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:07 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:07.778 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:07 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:49:07.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:07 smithi191 ceph-mon[36708]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:07.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:07 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:07.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:07 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:07.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:07 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:49:09.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:09 smithi012 ceph-mon[31541]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:09.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:09 smithi079 ceph-mon[36241]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:09 smithi191 ceph-mon[36708]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:10.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:49:10.551+0000 7f480dffb700 1 -- 172.21.15.79:0/3402645257 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4818062710 con 0x7f48000a2800 2024-03-17T23:49:10.553 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 1 on host 'smithi079' 2024-03-17T23:49:10.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:49:10.553+0000 7f481ee95700 1 -- 172.21.15.79:0/3402645257 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f48000a2800 msgr2=0x7f48000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:10.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:49:10.553+0000 7f481ee95700 1 --2- 172.21.15.79:0/3402645257 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f48000a2800 0x7f48000a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4814006fd0 tx=0x7f4814008040 comp rx=0 tx=0).stop 2024-03-17T23:49:10.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:49:10.553+0000 7f481ee95700 1 -- 172.21.15.79:0/3402645257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4818105760 msgr2=0x7f4818198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:10.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:49:10.553+0000 7f481ee95700 1 --2- 172.21.15.79:0/3402645257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4818105760 0x7f4818198ac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f480800b530 tx=0x7f480800b8f0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:49:10.553+0000 7f481ee95700 1 -- 172.21.15.79:0/3402645257 shutdown_connections 2024-03-17T23:49:10.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:49:10.553+0000 7f481ee95700 1 --2- 172.21.15.79:0/3402645257 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f48000a2800 0x7f48000a4cc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:49:10.553+0000 7f481ee95700 1 --2- 172.21.15.79:0/3402645257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4818108090 0x7f481819fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:49:10.553+0000 7f481ee95700 1 --2- 172.21.15.79:0/3402645257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4818102e30 0x7f4818198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:49:10.553+0000 7f481ee95700 1 --2- 172.21.15.79:0/3402645257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4818105760 0x7f4818198ac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:49:10.553+0000 7f481ee95700 1 -- 172.21.15.79:0/3402645257 >> 172.21.15.79:0/3402645257 conn(0x7f48180fc7c0 msgr2=0x7f4818103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:10.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:49:10.554+0000 7f481ee95700 1 -- 172.21.15.79:0/3402645257 shutdown_connections 2024-03-17T23:49:10.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:49:10.554+0000 7f481ee95700 1 -- 172.21.15.79:0/3402645257 wait complete. 2024-03-17T23:49:11.194 DEBUG:teuthology.orchestra.run.smithi079:osd.1> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.1.service 2024-03-17T23:49:11.197 INFO:tasks.cephadm:Deploying osd.2 on smithi191 with /dev/vg_nvme/lv_4... 2024-03-17T23:49:11.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-17T23:49:11.442 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-17T23:49:11.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:11 smithi191 ceph-mon[36708]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:11.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:11.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:11.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:11 smithi191 ceph-mon[36708]: from='osd.1 [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-17T23:49:11.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:11 smithi191 ceph-mon[36708]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-17T23:49:11.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:11.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:11.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:11 smithi079 ceph-mon[36241]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:11.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:11.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:11.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:11 smithi079 ceph-mon[36241]: from='osd.1 [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-17T23:49:11.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:11 smithi079 ceph-mon[36241]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-17T23:49:11.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:11.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:11.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:11 smithi012 ceph-mon[31541]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:11.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:11.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:11.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:11 smithi012 ceph-mon[31541]: from='osd.1 [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-17T23:49:11.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:11 smithi012 ceph-mon[31541]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-17T23:49:11.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:11.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:12.593 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:12 smithi191 ceph-mon[36708]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-17T23:49:12.593 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:12 smithi191 ceph-mon[36708]: osdmap e11: 2 total, 1 up, 2 in 2024-03-17T23:49:12.593 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:12.593 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:12 smithi191 ceph-mon[36708]: from='osd.1 [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:49:12.594 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:12 smithi191 ceph-mon[36708]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:49:12.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:12 smithi079 ceph-mon[36241]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-17T23:49:12.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:12 smithi079 ceph-mon[36241]: osdmap e11: 2 total, 1 up, 2 in 2024-03-17T23:49:12.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:12.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:12 smithi079 ceph-mon[36241]: from='osd.1 [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:49:12.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:12 smithi079 ceph-mon[36241]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:49:12.797 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 17 23:49:12 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1[40845]: 2024-03-17T23:49:12.538+0000 7f11b3063700 -1 osd.1 0 waiting for initial osdmap 2024-03-17T23:49:12.797 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 17 23:49:12 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1[40845]: 2024-03-17T23:49:12.544+0000 7f11aee56700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:49:12.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:12 smithi012 ceph-mon[31541]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-17T23:49:12.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:12 smithi012 ceph-mon[31541]: osdmap e11: 2 total, 1 up, 2 in 2024-03-17T23:49:12.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:12.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:12 smithi012 ceph-mon[31541]: from='osd.1 [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:49:12.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:12 smithi012 ceph-mon[31541]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:49:13.175 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-17T23:49:13.175 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-17T23:49:13.175 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-03-17T23:49:13.175 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-03-17T23:49:13.175 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115146 s, 911 MB/s 2024-03-17T23:49:13.175 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-03-17T23:49:13.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:13 smithi191 ceph-mon[36708]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:13.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:13 smithi191 ceph-mon[36708]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-17T23:49:13.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:13 smithi191 ceph-mon[36708]: osdmap e12: 2 total, 1 up, 2 in 2024-03-17T23:49:13.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:13 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:13.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:13 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:13.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi191:vg_nvme/lv_4 2024-03-17T23:49:13.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:13 smithi012 ceph-mon[31541]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:13.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:13 smithi012 ceph-mon[31541]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-17T23:49:13.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:13 smithi012 ceph-mon[31541]: osdmap e12: 2 total, 1 up, 2 in 2024-03-17T23:49:13.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:13.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:13.960 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:13 smithi079 ceph-mon[36241]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:13.960 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:13 smithi079 ceph-mon[36241]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-17T23:49:13.960 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:13 smithi079 ceph-mon[36241]: osdmap e12: 2 total, 1 up, 2 in 2024-03-17T23:49:13.960 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:13 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:13.960 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:13 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:14.067 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-17T23:49:14.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:14 smithi191 ceph-mon[36708]: purged_snaps scrub starts 2024-03-17T23:49:14.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:14 smithi191 ceph-mon[36708]: purged_snaps scrub ok 2024-03-17T23:49:14.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:14 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:14.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:14 smithi191 ceph-mon[36708]: osd.1 [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857] boot 2024-03-17T23:49:14.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:14 smithi191 ceph-mon[36708]: osdmap e13: 2 total, 2 up, 2 in 2024-03-17T23:49:14.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:14 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:14.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:14 smithi012 ceph-mon[31541]: purged_snaps scrub starts 2024-03-17T23:49:14.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:14 smithi012 ceph-mon[31541]: purged_snaps scrub ok 2024-03-17T23:49:14.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:14 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:14.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:14 smithi012 ceph-mon[31541]: osd.1 [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857] boot 2024-03-17T23:49:14.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:14 smithi012 ceph-mon[31541]: osdmap e13: 2 total, 2 up, 2 in 2024-03-17T23:49:14.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:14 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:15.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:14 smithi079 ceph-mon[36241]: purged_snaps scrub starts 2024-03-17T23:49:15.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:14 smithi079 ceph-mon[36241]: purged_snaps scrub ok 2024-03-17T23:49:15.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:14 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:15.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:14 smithi079 ceph-mon[36241]: osd.1 [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857] boot 2024-03-17T23:49:15.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:14 smithi079 ceph-mon[36241]: osdmap e13: 2 total, 2 up, 2 in 2024-03-17T23:49:15.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:14 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-17T23:49:15.600 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.597+0000 7f7913801700 1 -- 172.21.15.191:0/20250994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790c102300 msgr2=0x7f790c102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.601 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.597+0000 7f7913801700 1 --2- 172.21.15.191:0/20250994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790c102300 0x7f790c102720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7908009990 tx=0x7f7908009ca0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.601 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.597+0000 7f7913801700 1 -- 172.21.15.191:0/20250994 shutdown_connections 2024-03-17T23:49:15.601 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.597+0000 7f7913801700 1 --2- 172.21.15.191:0/20250994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f790c103ea0 0x7f790c10a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.601 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.597+0000 7f7913801700 1 --2- 172.21.15.191:0/20250994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790c102300 0x7f790c102720 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.601 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.598+0000 7f7913801700 1 --2- 172.21.15.191:0/20250994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f790c103500 0x7f790c103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.601 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.598+0000 7f7913801700 1 -- 172.21.15.191:0/20250994 >> 172.21.15.191:0/20250994 conn(0x7f790c0fd8e0 msgr2=0x7f790c0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:15.601 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.598+0000 7f7913801700 1 -- 172.21.15.191:0/20250994 shutdown_connections 2024-03-17T23:49:15.601 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.598+0000 7f7913801700 1 -- 172.21.15.191:0/20250994 wait complete. 2024-03-17T23:49:15.601 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.598+0000 7f7913801700 1 Processor -- start 2024-03-17T23:49:15.601 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.598+0000 7f7913801700 1 -- start start 2024-03-17T23:49:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.598+0000 7f7913801700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f790c102300 0x7f790c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.598+0000 7f7913801700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790c103500 0x7f790c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.598+0000 7f7913801700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f790c103ea0 0x7f790c19fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.598+0000 7f7913801700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f790c10df40 con 0x7f790c102300 2024-03-17T23:49:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.598+0000 7f7913801700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f790c10ddc0 con 0x7f790c103ea0 2024-03-17T23:49:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f7913801700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f790c10e0c0 con 0x7f790c103500 2024-03-17T23:49:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f7910d9c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790c103500 0x7f790c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f7910d9c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790c103500 0x7f790c198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:34766/0 (socket says 172.21.15.191:34766) 2024-03-17T23:49:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f7910d9c700 1 -- 172.21.15.191:0/622955941 learned_addr learned my addr 172.21.15.191:0/622955941 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-17T23:49:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f7911d9e700 1 --2- 172.21.15.191:0/622955941 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f790c103ea0 0x7f790c19fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f7910d9c700 1 -- 172.21.15.191:0/622955941 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f790c103ea0 msgr2=0x7f790c19fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f7910d9c700 1 --2- 172.21.15.191:0/622955941 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f790c103ea0 0x7f790c19fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f7910d9c700 1 -- 172.21.15.191:0/622955941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f790c102300 msgr2=0x7f790c1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.603 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f7910d9c700 1 --2- 172.21.15.191:0/622955941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f790c102300 0x7f790c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.603 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f7910d9c700 1 -- 172.21.15.191:0/622955941 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790c1a0210 con 0x7f790c103500 2024-03-17T23:49:15.603 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f7910d9c700 1 --2- 172.21.15.191:0/622955941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790c103500 0x7f790c198af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00d980 tx=0x7f78fc00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:15.603 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f79027fc700 1 -- 172.21.15.191:0/622955941 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc016070 con 0x7f790c103500 2024-03-17T23:49:15.603 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f7913801700 1 -- 172.21.15.191:0/622955941 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f790c1a0500 con 0x7f790c103500 2024-03-17T23:49:15.603 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f79027fc700 1 -- 172.21.15.191:0/622955941 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78fc00de70 con 0x7f790c103500 2024-03-17T23:49:15.603 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f79027fc700 1 -- 172.21.15.191:0/622955941 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc00f3c0 con 0x7f790c103500 2024-03-17T23:49:15.603 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.599+0000 7f7913801700 1 -- 172.21.15.191:0/622955941 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f790c1a0ac0 con 0x7f790c103500 2024-03-17T23:49:15.604 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.600+0000 7f7913801700 1 -- 172.21.15.191:0/622955941 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f790c068f90 con 0x7f790c103500 2024-03-17T23:49:15.604 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.602+0000 7f79027fc700 1 -- 172.21.15.191:0/622955941 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f78fc019070 con 0x7f790c103500 2024-03-17T23:49:15.604 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.602+0000 7f79027fc700 1 --2- 172.21.15.191:0/622955941 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f78f80a2800 0x7f78f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.604 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.603+0000 7f79027fc700 1 -- 172.21.15.191:0/622955941 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f78fc0ccd10 con 0x7f790c103500 2024-03-17T23:49:15.604 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.603+0000 7f791159d700 1 --2- 172.21.15.191:0/622955941 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f78f80a2800 0x7f78f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.605 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.604+0000 7f791159d700 1 --2- 172.21.15.191:0/622955941 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f78f80a2800 0x7f78f80a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7908005f50 tx=0x7f7908005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:15.607 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.606+0000 7f79027fc700 1 -- 172.21.15.191:0/622955941 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f78fc09b600 con 0x7f790c103500 2024-03-17T23:49:15.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:15.811+0000 7f7913801700 1 -- 172.21.15.191:0/622955941 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f790c062710 con 0x7f78f80a2800 2024-03-17T23:49:15.812 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: pgmap v73: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: osdmap e14: 2 total, 2 up, 2 in 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: Detected new or changed devices on smithi079 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi079 to 17080M 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:49:15.813 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:15.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: pgmap v73: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:15.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: osdmap e14: 2 total, 2 up, 2 in 2024-03-17T23:49:15.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: Detected new or changed devices on smithi079 2024-03-17T23:49:15.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:15.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:15.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:49:15.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi079 to 17080M 2024-03-17T23:49:15.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:15.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:15.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:49:15.923 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:15.923 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:49:15.923 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:15.923 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:49:15.923 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:16.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: pgmap v73: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-17T23:49:16.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: osdmap e14: 2 total, 2 up, 2 in 2024-03-17T23:49:16.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: Detected new or changed devices on smithi079 2024-03-17T23:49:16.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:16.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:16.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:49:16.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi079 to 17080M 2024-03-17T23:49:16.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:16.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:16.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:49:16.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:16.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:49:16.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:16.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:49:16.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:16.750 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:16 smithi191 ceph-mon[36708]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:49:16.750 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:16 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:49:16.750 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:16 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:49:16.750 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:16 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:16.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:16 smithi079 ceph-mon[36241]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:49:16.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:16 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:49:16.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:16 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:49:16.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:16 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:16.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:16 smithi012 ceph-mon[31541]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:49:16.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:16 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:49:16.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:16 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:49:16.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:16 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:17.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:17 smithi079 ceph-mon[36241]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:17.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:17 smithi191 ceph-mon[36708]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:17.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:17 smithi012 ceph-mon[31541]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:18.873 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:18 smithi191 ceph-mon[36708]: from='client.? 172.21.15.191:0/1170191484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24aaf834-2a6b-4bb0-986b-3fe9525ecacd"}]: dispatch 2024-03-17T23:49:18.873 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:18 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24aaf834-2a6b-4bb0-986b-3fe9525ecacd"}]: dispatch 2024-03-17T23:49:18.873 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:18 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24aaf834-2a6b-4bb0-986b-3fe9525ecacd"}]': finished 2024-03-17T23:49:18.873 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:18 smithi191 ceph-mon[36708]: osdmap e15: 3 total, 2 up, 3 in 2024-03-17T23:49:18.874 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:18 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:18.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:18 smithi012 ceph-mon[31541]: from='client.? 172.21.15.191:0/1170191484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24aaf834-2a6b-4bb0-986b-3fe9525ecacd"}]: dispatch 2024-03-17T23:49:18.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:18 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24aaf834-2a6b-4bb0-986b-3fe9525ecacd"}]: dispatch 2024-03-17T23:49:18.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:18 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24aaf834-2a6b-4bb0-986b-3fe9525ecacd"}]': finished 2024-03-17T23:49:18.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:18 smithi012 ceph-mon[31541]: osdmap e15: 3 total, 2 up, 3 in 2024-03-17T23:49:18.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:18 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:19.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:18 smithi079 ceph-mon[36241]: from='client.? 172.21.15.191:0/1170191484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24aaf834-2a6b-4bb0-986b-3fe9525ecacd"}]: dispatch 2024-03-17T23:49:19.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:18 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24aaf834-2a6b-4bb0-986b-3fe9525ecacd"}]: dispatch 2024-03-17T23:49:19.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:18 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24aaf834-2a6b-4bb0-986b-3fe9525ecacd"}]': finished 2024-03-17T23:49:19.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:18 smithi079 ceph-mon[36241]: osdmap e15: 3 total, 2 up, 3 in 2024-03-17T23:49:19.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:18 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:19 smithi012 ceph-mon[31541]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:19 smithi012 ceph-mon[31541]: from='client.? 172.21.15.191:0/134965827' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:49:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:19 smithi079 ceph-mon[36241]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:19 smithi079 ceph-mon[36241]: from='client.? 172.21.15.191:0/134965827' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:49:20.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:19 smithi191 ceph-mon[36708]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:20.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:19 smithi191 ceph-mon[36708]: from='client.? 172.21.15.191:0/134965827' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:49:21.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:21 smithi012 ceph-mon[31541]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:22.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:21 smithi079 ceph-mon[36241]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:22.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:21 smithi191 ceph-mon[36708]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:23.731 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:23 smithi191 ceph-mon[36708]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:23.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:23 smithi012 ceph-mon[31541]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:24.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:23 smithi079 ceph-mon[36241]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:25.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:25 smithi191 ceph-mon[36708]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:25.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-17T23:49:25.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:25.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:25 smithi012 ceph-mon[31541]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:25.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-17T23:49:25.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:26.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:25 smithi079 ceph-mon[36241]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:26.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-17T23:49:26.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:26.805 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:26 smithi191 ceph-mon[36708]: Deploying daemon osd.2 on smithi191 2024-03-17T23:49:26.916 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:26 smithi079 ceph-mon[36241]: Deploying daemon osd.2 on smithi191 2024-03-17T23:49:26.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:26 smithi012 ceph-mon[31541]: Deploying daemon osd.2 on smithi191 2024-03-17T23:49:27.677 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:27 smithi191 ceph-mon[36708]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:27.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:27 smithi079 ceph-mon[36241]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:27.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:27 smithi012 ceph-mon[31541]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:29.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:29 smithi012 ceph-mon[31541]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:30.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:29 smithi191 ceph-mon[36708]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:30.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:29 smithi079 ceph-mon[36241]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:31 smithi012 ceph-mon[31541]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:49:32.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:31 smithi079 ceph-mon[36241]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:32.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:32.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:32.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:49:32.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:31 smithi191 ceph-mon[36708]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:32.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:32.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:32.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:49:34.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:33 smithi012 ceph-mon[31541]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:34.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:34.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:34.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:34.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:49:34.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:34.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:33 smithi191 ceph-mon[36708]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:34.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:34.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:34.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:34.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:49:34.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:34.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:33 smithi079 ceph-mon[36241]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:34.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:34.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:34.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:34.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:49:34.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:34.598 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:34.596+0000 7f79027fc700 1 -- 172.21.15.191:0/622955941 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f790c062710 con 0x7f78f80a2800 2024-03-17T23:49:34.598 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 2 on host 'smithi191' 2024-03-17T23:49:34.599 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:34.598+0000 7f7913801700 1 -- 172.21.15.191:0/622955941 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f78f80a2800 msgr2=0x7f78f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.599 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:34.598+0000 7f7913801700 1 --2- 172.21.15.191:0/622955941 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f78f80a2800 0x7f78f80a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7908005f50 tx=0x7f7908005e80 comp rx=0 tx=0).stop 2024-03-17T23:49:34.599 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:34.598+0000 7f7913801700 1 -- 172.21.15.191:0/622955941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790c103500 msgr2=0x7f790c198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.599 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:34.598+0000 7f7913801700 1 --2- 172.21.15.191:0/622955941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790c103500 0x7f790c198af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00d980 tx=0x7f78fc00dc90 comp rx=0 tx=0).stop 2024-03-17T23:49:34.599 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:34.598+0000 7f7913801700 1 -- 172.21.15.191:0/622955941 shutdown_connections 2024-03-17T23:49:34.599 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:34.598+0000 7f7913801700 1 --2- 172.21.15.191:0/622955941 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f78f80a2800 0x7f78f80a4cc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.600 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:34.598+0000 7f7913801700 1 --2- 172.21.15.191:0/622955941 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f790c103ea0 0x7f790c19fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.600 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:34.598+0000 7f7913801700 1 --2- 172.21.15.191:0/622955941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f790c102300 0x7f790c1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.600 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:34.598+0000 7f7913801700 1 --2- 172.21.15.191:0/622955941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790c103500 0x7f790c198af0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.600 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:34.598+0000 7f7913801700 1 -- 172.21.15.191:0/622955941 >> 172.21.15.191:0/622955941 conn(0x7f790c0fd8e0 msgr2=0x7f790c1068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:34.600 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:34.598+0000 7f7913801700 1 -- 172.21.15.191:0/622955941 shutdown_connections 2024-03-17T23:49:34.600 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:49:34.598+0000 7f7913801700 1 -- 172.21.15.191:0/622955941 wait complete. 2024-03-17T23:49:35.287 DEBUG:teuthology.orchestra.run.smithi191:osd.2> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.2.service 2024-03-17T23:49:35.289 INFO:tasks.cephadm:Deploying osd.3 on smithi012 with /dev/vg_nvme/lv_3... 2024-03-17T23:49:35.289 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-17T23:49:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:35 smithi012 ceph-mon[31541]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:49:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:35 smithi012 ceph-mon[31541]: from='osd.2 [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-17T23:49:35.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:35 smithi012 ceph-mon[31541]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-17T23:49:35.746 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:49:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:35 smithi079 ceph-mon[36241]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:49:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:35 smithi079 ceph-mon[36241]: from='osd.2 [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-17T23:49:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:35 smithi079 ceph-mon[36241]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-17T23:49:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:35 smithi191 ceph-mon[36708]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:49:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:35 smithi191 ceph-mon[36708]: from='osd.2 [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-17T23:49:35.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:35 smithi191 ceph-mon[36708]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-17T23:49:36.770 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:36 smithi012 ceph-mon[31541]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-17T23:49:36.770 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:36 smithi012 ceph-mon[31541]: from='osd.2 [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:49:36.770 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:36 smithi012 ceph-mon[31541]: osdmap e16: 3 total, 2 up, 3 in 2024-03-17T23:49:36.770 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:36 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:36.770 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:36 smithi012 ceph-mon[31541]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:49:36.830 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:36 smithi191 ceph-mon[36708]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-17T23:49:36.831 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:36 smithi191 ceph-mon[36708]: from='osd.2 [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:49:36.831 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:36 smithi191 ceph-mon[36708]: osdmap e16: 3 total, 2 up, 3 in 2024-03-17T23:49:36.831 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:36 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:36.831 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:36 smithi191 ceph-mon[36708]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:49:36.831 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 17 23:49:36 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2[41866]: 2024-03-17T23:49:36.557+0000 7f6c6e0a6700 -1 osd.2 0 waiting for initial osdmap 2024-03-17T23:49:36.832 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 17 23:49:36 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2[41866]: 2024-03-17T23:49:36.561+0000 7f6c69e99700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:49:37.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:36 smithi079 ceph-mon[36241]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-17T23:49:37.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:36 smithi079 ceph-mon[36241]: from='osd.2 [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:49:37.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:36 smithi079 ceph-mon[36241]: osdmap e16: 3 total, 2 up, 3 in 2024-03-17T23:49:37.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:36 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:37.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:36 smithi079 ceph-mon[36241]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:49:37.640 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:37 smithi191 ceph-mon[36708]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:37.641 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:37 smithi191 ceph-mon[36708]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-17T23:49:37.641 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:37 smithi191 ceph-mon[36708]: osdmap e17: 3 total, 2 up, 3 in 2024-03-17T23:49:37.641 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:37 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:37.641 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:37 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:37.734 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:37 smithi012 ceph-mon[31541]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:37.734 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:37 smithi012 ceph-mon[31541]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-17T23:49:37.734 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:37 smithi012 ceph-mon[31541]: osdmap e17: 3 total, 2 up, 3 in 2024-03-17T23:49:37.734 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:37.734 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:38.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:37 smithi079 ceph-mon[36241]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:38.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:37 smithi079 ceph-mon[36241]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-17T23:49:38.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:37 smithi079 ceph-mon[36241]: osdmap e17: 3 total, 2 up, 3 in 2024-03-17T23:49:38.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:37 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:38.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:37 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:38.712 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-17T23:49:38.713 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-17T23:49:38.713 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-03-17T23:49:38.713 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-03-17T23:49:38.713 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0678863 s, 154 MB/s 2024-03-17T23:49:38.713 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-03-17T23:49:38.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:38 smithi012 ceph-mon[31541]: purged_snaps scrub starts 2024-03-17T23:49:38.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:38 smithi012 ceph-mon[31541]: purged_snaps scrub ok 2024-03-17T23:49:38.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:38 smithi012 ceph-mon[31541]: osd.2 [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993] boot 2024-03-17T23:49:38.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:38 smithi012 ceph-mon[31541]: osdmap e18: 3 total, 3 up, 3 in 2024-03-17T23:49:38.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:38 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:38.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:38 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-17T23:49:39.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:38 smithi079 ceph-mon[36241]: purged_snaps scrub starts 2024-03-17T23:49:39.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:38 smithi079 ceph-mon[36241]: purged_snaps scrub ok 2024-03-17T23:49:39.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:38 smithi079 ceph-mon[36241]: osd.2 [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993] boot 2024-03-17T23:49:39.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:38 smithi079 ceph-mon[36241]: osdmap e18: 3 total, 3 up, 3 in 2024-03-17T23:49:39.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:38 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:39.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:38 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-17T23:49:39.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:38 smithi191 ceph-mon[36708]: purged_snaps scrub starts 2024-03-17T23:49:39.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:38 smithi191 ceph-mon[36708]: purged_snaps scrub ok 2024-03-17T23:49:39.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:38 smithi191 ceph-mon[36708]: osd.2 [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993] boot 2024-03-17T23:49:39.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:38 smithi191 ceph-mon[36708]: osdmap e18: 3 total, 3 up, 3 in 2024-03-17T23:49:39.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:38 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-17T23:49:39.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:38 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-17T23:49:39.653 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi012:vg_nvme/lv_3 2024-03-17T23:49:39.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:39 smithi012 ceph-mon[31541]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:39.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:39 smithi012 ceph-mon[31541]: Detected new or changed devices on smithi191 2024-03-17T23:49:39.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-17T23:49:39.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:39.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:39 smithi012 ceph-mon[31541]: osdmap e19: 3 total, 3 up, 3 in 2024-03-17T23:49:39.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-17T23:49:39.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:39.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:49:39.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:39 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi191 to 17080M 2024-03-17T23:49:39.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:39.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:39.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:49:39.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:40.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:39 smithi079 ceph-mon[36241]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:40.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:39 smithi079 ceph-mon[36241]: Detected new or changed devices on smithi191 2024-03-17T23:49:40.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-17T23:49:40.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:40.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:39 smithi079 ceph-mon[36241]: osdmap e19: 3 total, 3 up, 3 in 2024-03-17T23:49:40.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-17T23:49:40.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:40.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:49:40.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:39 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi191 to 17080M 2024-03-17T23:49:40.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:40.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:40.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:49:40.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:40.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:39 smithi191 ceph-mon[36708]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-17T23:49:40.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:39 smithi191 ceph-mon[36708]: Detected new or changed devices on smithi191 2024-03-17T23:49:40.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-17T23:49:40.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:40.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:39 smithi191 ceph-mon[36708]: osdmap e19: 3 total, 3 up, 3 in 2024-03-17T23:49:40.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-17T23:49:40.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:40.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:49:40.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:39 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi191 to 17080M 2024-03-17T23:49:40.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:40.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:40.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:49:40.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:40.122 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:49:40.557 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 17 23:49:40 smithi012 sudo[48437]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-17T23:49:40.557 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 17 23:49:40 smithi012 sudo[48437]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-17T23:49:40.557 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 17 23:49:40 smithi012 sudo[48437]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-17T23:49:40.557 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 17 23:49:40 smithi012 sudo[48437]: pam_unix(sudo:session): session closed for user root 2024-03-17T23:49:40.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:40 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-17T23:49:40.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:40 smithi012 ceph-mon[31541]: osdmap e20: 3 total, 3 up, 3 in 2024-03-17T23:49:41.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:40 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-17T23:49:41.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:40 smithi079 ceph-mon[36241]: osdmap e20: 3 total, 3 up, 3 in 2024-03-17T23:49:41.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:40 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-17T23:49:41.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:40 smithi191 ceph-mon[36708]: osdmap e20: 3 total, 3 up, 3 in 2024-03-17T23:49:41.299 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 17 23:49:41 smithi012 sudo[48456]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-17T23:49:41.299 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 17 23:49:41 smithi012 sudo[48456]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-17T23:49:41.299 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 17 23:49:41 smithi012 sudo[48456]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-17T23:49:41.299 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 17 23:49:41 smithi012 sudo[48456]: pam_unix(sudo:session): session closed for user root 2024-03-17T23:49:41.570 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 17 23:49:41 smithi079 sudo[45005]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-17T23:49:41.570 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 17 23:49:41 smithi079 sudo[45005]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-17T23:49:41.570 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 17 23:49:41 smithi079 sudo[45005]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-17T23:49:41.570 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 17 23:49:41 smithi079 sudo[45005]: pam_unix(sudo:session): session closed for user root 2024-03-17T23:49:41.909 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:41 smithi012 ceph-mon[31541]: pgmap v93: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:41.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:41 smithi012 ceph-mon[31541]: osdmap e21: 3 total, 3 up, 3 in 2024-03-17T23:49:41.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:41 smithi012 ceph-mon[31541]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-17T23:49:41.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:41 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:41.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:41 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:42.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:41 smithi079 ceph-mon[36241]: pgmap v93: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:42.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:41 smithi079 ceph-mon[36241]: osdmap e21: 3 total, 3 up, 3 in 2024-03-17T23:49:42.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:41 smithi079 ceph-mon[36241]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-17T23:49:42.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:41 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:42.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:41 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:42.046 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 17 23:49:41 smithi079 sudo[45008]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-17T23:49:42.046 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 17 23:49:41 smithi079 sudo[45008]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-17T23:49:42.047 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 17 23:49:41 smithi079 sudo[45008]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-17T23:49:42.047 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 17 23:49:41 smithi079 sudo[45008]: pam_unix(sudo:session): session closed for user root 2024-03-17T23:49:42.091 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 17 23:49:41 smithi191 sudo[45511]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-17T23:49:42.091 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 17 23:49:41 smithi191 sudo[45511]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-17T23:49:42.091 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 17 23:49:41 smithi191 sudo[45511]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-17T23:49:42.091 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 17 23:49:41 smithi191 sudo[45511]: pam_unix(sudo:session): session closed for user root 2024-03-17T23:49:42.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:41 smithi191 ceph-mon[36708]: pgmap v93: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:42.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:41 smithi191 ceph-mon[36708]: osdmap e21: 3 total, 3 up, 3 in 2024-03-17T23:49:42.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:41 smithi191 ceph-mon[36708]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-17T23:49:42.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:41 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:42.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:41 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:42.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.460+0000 7f7a74c00700 1 -- 172.21.15.12:0/714514366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a70107fd0 msgr2=0x7f7a7010a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:42.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.460+0000 7f7a74c00700 1 --2- 172.21.15.12:0/714514366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a70107fd0 0x7f7a7010a3c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800b0a0 tx=0x7f7a6800b3b0 comp rx=0 tx=0).stop 2024-03-17T23:49:42.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.460+0000 7f7a74c00700 1 -- 172.21.15.12:0/714514366 shutdown_connections 2024-03-17T23:49:42.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.460+0000 7f7a74c00700 1 --2- 172.21.15.12:0/714514366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a70107fd0 0x7f7a7010a3c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:42.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.460+0000 7f7a74c00700 1 --2- 172.21.15.12:0/714514366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a70102d70 0x7f7a70105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:42.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.460+0000 7f7a74c00700 1 --2- 172.21.15.12:0/714514366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a701056a0 0x7f7a70107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:42.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.460+0000 7f7a74c00700 1 -- 172.21.15.12:0/714514366 >> 172.21.15.12:0/714514366 conn(0x7f7a700fc6a0 msgr2=0x7f7a700feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.460+0000 7f7a74c00700 1 -- 172.21.15.12:0/714514366 shutdown_connections 2024-03-17T23:49:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.460+0000 7f7a74c00700 1 -- 172.21.15.12:0/714514366 wait complete. 2024-03-17T23:49:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.461+0000 7f7a74c00700 1 Processor -- start 2024-03-17T23:49:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.462+0000 7f7a74c00700 1 -- start start 2024-03-17T23:49:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.462+0000 7f7a74c00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a70102d70 0x7f7a701026d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.462+0000 7f7a74c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a701056a0 0x7f7a70100d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.462+0000 7f7a74c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a70107fd0 0x7f7a70101260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.462+0000 7f7a74c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7010de90 con 0x7f7a701056a0 2024-03-17T23:49:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.462+0000 7f7a74c00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7010dd10 con 0x7f7a70102d70 2024-03-17T23:49:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.462+0000 7f7a74c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7010e010 con 0x7f7a70107fd0 2024-03-17T23:49:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.462+0000 7f7a6f400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a70102d70 0x7f7a701026d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.462+0000 7f7a6ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a701056a0 0x7f7a70100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.462+0000 7f7a6fe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a70107fd0 0x7f7a70101260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.463+0000 7f7a6f400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a70102d70 0x7f7a701026d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:53082/0 (socket says 172.21.15.12:53082) 2024-03-17T23:49:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.463+0000 7f7a6f400700 1 -- 172.21.15.12:0/2530501687 learned_addr learned my addr 172.21.15.12:0/2530501687 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:49:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.463+0000 7f7a6f400700 1 -- 172.21.15.12:0/2530501687 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a70107fd0 msgr2=0x7f7a70101260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.463+0000 7f7a6f400700 1 --2- 172.21.15.12:0/2530501687 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a70107fd0 0x7f7a70101260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.463+0000 7f7a6f400700 1 -- 172.21.15.12:0/2530501687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a701056a0 msgr2=0x7f7a70100d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.463+0000 7f7a6f400700 1 --2- 172.21.15.12:0/2530501687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a701056a0 0x7f7a70100d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.463+0000 7f7a6f400700 1 -- 172.21.15.12:0/2530501687 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a70101b10 con 0x7f7a70102d70 2024-03-17T23:49:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.463+0000 7f7a6f400700 1 --2- 172.21.15.12:0/2530501687 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a70102d70 0x7f7a701026d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000b530 tx=0x7f7a6000b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.464+0000 7f7a67e00700 1 -- 172.21.15.12:0/2530501687 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a60015020 con 0x7f7a70102d70 2024-03-17T23:49:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.464+0000 7f7a74c00700 1 -- 172.21.15.12:0/2530501687 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a701a25d0 con 0x7f7a70102d70 2024-03-17T23:49:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.464+0000 7f7a67e00700 1 -- 172.21.15.12:0/2530501687 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a60011d10 con 0x7f7a70102d70 2024-03-17T23:49:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.464+0000 7f7a67e00700 1 -- 172.21.15.12:0/2530501687 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a6000f2e0 con 0x7f7a70102d70 2024-03-17T23:49:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.464+0000 7f7a74c00700 1 -- 172.21.15.12:0/2530501687 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a701a2b10 con 0x7f7a70102d70 2024-03-17T23:49:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.465+0000 7f7a74c00700 1 -- 172.21.15.12:0/2530501687 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a70068f90 con 0x7f7a70102d70 2024-03-17T23:49:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.467+0000 7f7a67e00700 1 -- 172.21.15.12:0/2530501687 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7a60018020 con 0x7f7a70102d70 2024-03-17T23:49:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.468+0000 7f7a67e00700 1 --2- 172.21.15.12:0/2530501687 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7a540a2890 0x7f7a540a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:42.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.468+0000 7f7a67e00700 1 -- 172.21.15.12:0/2530501687 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f7a600cce10 con 0x7f7a70102d70 2024-03-17T23:49:42.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.468+0000 7f7a6ea00700 1 --2- 172.21.15.12:0/2530501687 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7a540a2890 0x7f7a540a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:42.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.469+0000 7f7a6ea00700 1 --2- 172.21.15.12:0/2530501687 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7a540a2890 0x7f7a540a4d50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009c60 tx=0x7f7a5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:42.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.473+0000 7f7a67e00700 1 -- 172.21.15.12:0/2530501687 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7a600d0030 con 0x7f7a70102d70 2024-03-17T23:49:42.591 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 17 23:49:42 smithi191 sudo[45514]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-17T23:49:42.591 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 17 23:49:42 smithi191 sudo[45514]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-17T23:49:42.591 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 17 23:49:42 smithi191 sudo[45514]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-17T23:49:42.591 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 17 23:49:42 smithi191 sudo[45514]: pam_unix(sudo:session): session closed for user root 2024-03-17T23:49:42.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:49:42.719+0000 7f7a74c00700 1 -- 172.21.15.12:0/2530501687 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f7a70062710 con 0x7f7a540a2890 2024-03-17T23:49:42.720 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:42 smithi012 sudo[48510]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-17T23:49:42.720 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:42 smithi012 sudo[48510]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-17T23:49:42.720 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:42 smithi012 sudo[48510]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-17T23:49:43.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:42 smithi012 sudo[48510]: pam_unix(sudo:session): session closed for user root 2024-03-17T23:49:43.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:43 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:43.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:43 smithi079 ceph-mon[36241]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-17T23:49:43.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:43 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:43.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:43 smithi191 ceph-mon[36708]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-17T23:49:43.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:43 smithi191 sudo[45517]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-17T23:49:43.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:43 smithi191 sudo[45517]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-17T23:49:43.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:43 smithi191 sudo[45517]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-17T23:49:43.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:43 smithi191 sudo[45517]: pam_unix(sudo:session): session closed for user root 2024-03-17T23:49:43.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:43 smithi012 sudo[48514]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-17T23:49:43.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:43 smithi012 sudo[48514]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-17T23:49:43.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:43 smithi012 sudo[48514]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-17T23:49:43.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:43 smithi012 sudo[48514]: pam_unix(sudo:session): session closed for user root 2024-03-17T23:49:43.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:43 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:49:43.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:43 smithi012 ceph-mon[31541]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-17T23:49:43.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:43 smithi191 sudo[45520]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-17T23:49:43.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:43 smithi191 sudo[45520]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-17T23:49:43.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:43 smithi191 sudo[45520]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-17T23:49:43.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:43 smithi191 sudo[45520]: pam_unix(sudo:session): session closed for user root 2024-03-17T23:49:44.286 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 sudo[45011]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-17T23:49:44.286 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 sudo[45011]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-17T23:49:44.286 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 sudo[45011]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-17T23:49:44.287 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 sudo[45011]: pam_unix(sudo:session): session closed for user root 2024-03-17T23:49:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 sudo[45014]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-17T23:49:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 sudo[45014]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-17T23:49:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 sudo[45014]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-17T23:49:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 sudo[45014]: pam_unix(sudo:session): session closed for user root 2024-03-17T23:49:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:49:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-17T23:49:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:49:44.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:44 smithi079 ceph-mon[36241]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-17T23:49:44.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:44.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:49:44.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-17T23:49:44.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:49:44.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:49:44.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:49:44.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:49:44.562 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-17T23:49:44.562 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:49:44.562 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:49:44.562 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:49:44.562 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:49:44.563 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:44.563 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-17T23:49:44.563 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-17T23:49:44.563 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:49:44.563 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:49:44.564 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:49:44.564 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:44 smithi012 ceph-mon[31541]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-17T23:49:44.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:44.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:49:44.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-17T23:49:44.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:49:44.593 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:49:44.593 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:49:44.593 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:49:44.593 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-17T23:49:44.594 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:49:44.594 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:49:44.594 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:49:44.594 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:49:44.594 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:44.595 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-17T23:49:44.595 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-17T23:49:44.595 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-17T23:49:44.596 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-17T23:49:44.596 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-17T23:49:44.596 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:44 smithi191 ceph-mon[36708]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-17T23:49:45.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:45 smithi012 ceph-mon[31541]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:45.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:45 smithi191 ceph-mon[36708]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:45.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:45 smithi079 ceph-mon[36241]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:47.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:47 smithi012 ceph-mon[31541]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:47.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:47 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2648093455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba4c0892-f619-45d4-a91a-00a2b0cc3d78"}]: dispatch 2024-03-17T23:49:47.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:47 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba4c0892-f619-45d4-a91a-00a2b0cc3d78"}]: dispatch 2024-03-17T23:49:47.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:47 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba4c0892-f619-45d4-a91a-00a2b0cc3d78"}]': finished 2024-03-17T23:49:47.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:47 smithi012 ceph-mon[31541]: osdmap e22: 4 total, 3 up, 4 in 2024-03-17T23:49:47.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:47 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:49:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:47 smithi191 ceph-mon[36708]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:47 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2648093455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba4c0892-f619-45d4-a91a-00a2b0cc3d78"}]: dispatch 2024-03-17T23:49:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:47 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba4c0892-f619-45d4-a91a-00a2b0cc3d78"}]: dispatch 2024-03-17T23:49:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:47 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba4c0892-f619-45d4-a91a-00a2b0cc3d78"}]': finished 2024-03-17T23:49:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:47 smithi191 ceph-mon[36708]: osdmap e22: 4 total, 3 up, 4 in 2024-03-17T23:49:47.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:47 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:49:47.623 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:47 smithi079 ceph-mon[36241]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:47.623 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:47 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2648093455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba4c0892-f619-45d4-a91a-00a2b0cc3d78"}]: dispatch 2024-03-17T23:49:47.623 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:47 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba4c0892-f619-45d4-a91a-00a2b0cc3d78"}]: dispatch 2024-03-17T23:49:47.623 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:47 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba4c0892-f619-45d4-a91a-00a2b0cc3d78"}]': finished 2024-03-17T23:49:47.623 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:47 smithi079 ceph-mon[36241]: osdmap e22: 4 total, 3 up, 4 in 2024-03-17T23:49:47.623 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:47 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:49:48.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:48 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2586974093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:49:48.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:48 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2586974093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:49:48.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:48 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2586974093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:49:49.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:49 smithi191 ceph-mon[36708]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:49.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:49 smithi012 ceph-mon[31541]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:49.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:49 smithi079 ceph-mon[36241]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:51.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:51 smithi191 ceph-mon[36708]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:51.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:51 smithi012 ceph-mon[31541]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:51.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:51 smithi079 ceph-mon[36241]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:53.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:53 smithi191 ceph-mon[36708]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:53.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:53 smithi012 ceph-mon[31541]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:53.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:53 smithi079 ceph-mon[36241]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:55.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:55 smithi191 ceph-mon[36708]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:55.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:55 smithi012 ceph-mon[31541]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:55.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:55 smithi079 ceph-mon[36241]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:57.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:57 smithi079 ceph-mon[36241]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:57.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-17T23:49:57.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:57.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:57 smithi079 ceph-mon[36241]: Deploying daemon osd.3 on smithi012 2024-03-17T23:49:57.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:57 smithi012 ceph-mon[31541]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:57.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-17T23:49:57.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:57.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:57 smithi012 ceph-mon[31541]: Deploying daemon osd.3 on smithi012 2024-03-17T23:49:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:57 smithi191 ceph-mon[36708]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-17T23:49:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:49:57.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:57 smithi191 ceph-mon[36708]: Deploying daemon osd.3 on smithi012 2024-03-17T23:49:59.393 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:49:59 smithi012 ceph-mon[31541]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:59.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:49:59 smithi079 ceph-mon[36241]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:49:59.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:49:59 smithi191 ceph-mon[36708]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:00.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:00 smithi012 ceph-mon[31541]: overall HEALTH_OK 2024-03-17T23:50:00.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:00 smithi079 ceph-mon[36241]: overall HEALTH_OK 2024-03-17T23:50:00.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:00 smithi191 ceph-mon[36708]: overall HEALTH_OK 2024-03-17T23:50:01.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:01 smithi012 ceph-mon[31541]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:01.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:01 smithi079 ceph-mon[36241]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:01.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:01 smithi191 ceph-mon[36708]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:03.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:03 smithi012 ceph-mon[31541]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:03.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:03 smithi079 ceph-mon[36241]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:03.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:03 smithi191 ceph-mon[36708]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:06.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:05 smithi079 ceph-mon[36241]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:06.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:06.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:06.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:06.078 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:05 smithi012 ceph-mon[31541]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:06.078 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:06.078 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:06.078 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:06.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:05 smithi191 ceph-mon[36708]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:06.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:06.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:06.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:08.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:07 smithi079 ceph-mon[36241]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:08.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:07 smithi191 ceph-mon[36708]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:08.098 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:07 smithi012 ceph-mon[31541]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:09.617 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:09 smithi012 ceph-mon[31541]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:09.618 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:09.618 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:09.618 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:09 smithi012 ceph-mon[31541]: from='osd.3 [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-17T23:50:09.618 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:09 smithi012 ceph-mon[31541]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-17T23:50:09.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:50:09.698+0000 7f7a67e00700 1 -- 172.21.15.12:0/2530501687 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7a70062710 con 0x7f7a540a2890 2024-03-17T23:50:09.700 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 3 on host 'smithi012' 2024-03-17T23:50:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:50:09.702+0000 7f7a74c00700 1 -- 172.21.15.12:0/2530501687 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7a540a2890 msgr2=0x7f7a540a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:50:09.702+0000 7f7a74c00700 1 --2- 172.21.15.12:0/2530501687 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7a540a2890 0x7f7a540a4d50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009c60 tx=0x7f7a5c009260 comp rx=0 tx=0).stop 2024-03-17T23:50:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:50:09.702+0000 7f7a74c00700 1 -- 172.21.15.12:0/2530501687 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a70102d70 msgr2=0x7f7a701026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:50:09.702+0000 7f7a74c00700 1 --2- 172.21.15.12:0/2530501687 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a70102d70 0x7f7a701026d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000b530 tx=0x7f7a6000b840 comp rx=0 tx=0).stop 2024-03-17T23:50:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:50:09.702+0000 7f7a74c00700 1 -- 172.21.15.12:0/2530501687 shutdown_connections 2024-03-17T23:50:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:50:09.702+0000 7f7a74c00700 1 --2- 172.21.15.12:0/2530501687 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7a540a2890 0x7f7a540a4d50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:50:09.702+0000 7f7a74c00700 1 --2- 172.21.15.12:0/2530501687 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a70107fd0 0x7f7a70101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:50:09.702+0000 7f7a74c00700 1 --2- 172.21.15.12:0/2530501687 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a70102d70 0x7f7a701026d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:50:09.702+0000 7f7a74c00700 1 --2- 172.21.15.12:0/2530501687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a701056a0 0x7f7a70100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:50:09.702+0000 7f7a74c00700 1 -- 172.21.15.12:0/2530501687 >> 172.21.15.12:0/2530501687 conn(0x7f7a700fc6a0 msgr2=0x7f7a701036b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:50:09.702+0000 7f7a74c00700 1 -- 172.21.15.12:0/2530501687 shutdown_connections 2024-03-17T23:50:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:50:09.702+0000 7f7a74c00700 1 -- 172.21.15.12:0/2530501687 wait complete. 2024-03-17T23:50:09.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:09 smithi079 ceph-mon[36241]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:09.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:09 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:09.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:09 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:09.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:09 smithi079 ceph-mon[36241]: from='osd.3 [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-17T23:50:09.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:09 smithi079 ceph-mon[36241]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-17T23:50:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:09 smithi191 ceph-mon[36708]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:09 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:09 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:09 smithi191 ceph-mon[36708]: from='osd.3 [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-17T23:50:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:09 smithi191 ceph-mon[36708]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-17T23:50:10.470 DEBUG:teuthology.orchestra.run.smithi012:osd.3> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.3.service 2024-03-17T23:50:10.475 INFO:tasks.cephadm:Deploying osd.4 on smithi079 with /dev/vg_nvme/lv_3... 2024-03-17T23:50:10.476 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-17T23:50:10.735 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.b/config 2024-03-17T23:50:10.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:10 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:10.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:10 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:10.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:10 smithi079 ceph-mon[36241]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-17T23:50:10.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:10 smithi079 ceph-mon[36241]: from='osd.3 [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:50:10.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:10 smithi079 ceph-mon[36241]: osdmap e23: 4 total, 3 up, 4 in 2024-03-17T23:50:10.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:10 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:50:10.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:10 smithi079 ceph-mon[36241]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:50:10.925 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:10 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:10.925 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:10 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:10.925 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:10 smithi012 ceph-mon[31541]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-17T23:50:10.925 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:10 smithi012 ceph-mon[31541]: from='osd.3 [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:50:10.925 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:10 smithi012 ceph-mon[31541]: osdmap e23: 4 total, 3 up, 4 in 2024-03-17T23:50:10.925 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:10 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:50:10.925 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:10 smithi012 ceph-mon[31541]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:50:11.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:10 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:11.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:10 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:11.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:10 smithi191 ceph-mon[36708]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-17T23:50:11.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:10 smithi191 ceph-mon[36708]: from='osd.3 [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:50:11.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:10 smithi191 ceph-mon[36708]: osdmap e23: 4 total, 3 up, 4 in 2024-03-17T23:50:11.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:10 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:50:11.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:10 smithi191 ceph-mon[36708]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:50:11.822 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 17 23:50:11 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3[50772]: 2024-03-17T23:50:11.448+0000 7fe677c00700 -1 osd.3 0 waiting for initial osdmap 2024-03-17T23:50:11.822 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 17 23:50:11 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3[50772]: 2024-03-17T23:50:11.465+0000 7fe66d000700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:50:12.073 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:11 smithi079 ceph-mon[36241]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:12.074 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:11 smithi079 ceph-mon[36241]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-17T23:50:12.074 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:11 smithi079 ceph-mon[36241]: osdmap e24: 4 total, 3 up, 4 in 2024-03-17T23:50:12.074 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:50:12.074 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:50:12.074 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:11 smithi079 ceph-mon[36241]: from='osd.3 ' entity='osd.3' 2024-03-17T23:50:12.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:11 smithi191 ceph-mon[36708]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:12.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:11 smithi191 ceph-mon[36708]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-17T23:50:12.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:11 smithi191 ceph-mon[36708]: osdmap e24: 4 total, 3 up, 4 in 2024-03-17T23:50:12.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:50:12.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:50:12.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:11 smithi191 ceph-mon[36708]: from='osd.3 ' entity='osd.3' 2024-03-17T23:50:12.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:11 smithi012 ceph-mon[31541]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-17T23:50:12.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:11 smithi012 ceph-mon[31541]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-17T23:50:12.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:11 smithi012 ceph-mon[31541]: osdmap e24: 4 total, 3 up, 4 in 2024-03-17T23:50:12.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:50:12.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:50:12.162 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:11 smithi012 ceph-mon[31541]: from='osd.3 ' entity='osd.3' 2024-03-17T23:50:12.695 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-17T23:50:12.695 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-17T23:50:12.695 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-03-17T23:50:12.696 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-03-17T23:50:12.696 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123769 s, 847 MB/s 2024-03-17T23:50:12.696 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-03-17T23:50:13.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:12 smithi079 ceph-mon[36241]: purged_snaps scrub starts 2024-03-17T23:50:13.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:12 smithi079 ceph-mon[36241]: purged_snaps scrub ok 2024-03-17T23:50:13.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:12 smithi079 ceph-mon[36241]: osd.3 [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575] boot 2024-03-17T23:50:13.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:12 smithi079 ceph-mon[36241]: osdmap e25: 4 total, 4 up, 4 in 2024-03-17T23:50:13.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:50:13.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:12 smithi079 ceph-mon[36241]: osdmap e26: 4 total, 4 up, 4 in 2024-03-17T23:50:13.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:12 smithi191 ceph-mon[36708]: purged_snaps scrub starts 2024-03-17T23:50:13.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:12 smithi191 ceph-mon[36708]: purged_snaps scrub ok 2024-03-17T23:50:13.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:12 smithi191 ceph-mon[36708]: osd.3 [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575] boot 2024-03-17T23:50:13.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:12 smithi191 ceph-mon[36708]: osdmap e25: 4 total, 4 up, 4 in 2024-03-17T23:50:13.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:50:13.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:12 smithi191 ceph-mon[36708]: osdmap e26: 4 total, 4 up, 4 in 2024-03-17T23:50:13.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:12 smithi012 ceph-mon[31541]: purged_snaps scrub starts 2024-03-17T23:50:13.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:12 smithi012 ceph-mon[31541]: purged_snaps scrub ok 2024-03-17T23:50:13.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:12 smithi012 ceph-mon[31541]: osd.3 [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575] boot 2024-03-17T23:50:13.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:12 smithi012 ceph-mon[31541]: osdmap e25: 4 total, 4 up, 4 in 2024-03-17T23:50:13.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-17T23:50:13.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:12 smithi012 ceph-mon[31541]: osdmap e26: 4 total, 4 up, 4 in 2024-03-17T23:50:13.407 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi079:vg_nvme/lv_3 2024-03-17T23:50:13.645 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.b/config 2024-03-17T23:50:13.966 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:13 smithi079 ceph-mon[36241]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:13.967 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:13 smithi079 ceph-mon[36241]: osdmap e27: 4 total, 4 up, 4 in 2024-03-17T23:50:14.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:13 smithi191 ceph-mon[36708]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:14.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:13 smithi191 ceph-mon[36708]: osdmap e27: 4 total, 4 up, 4 in 2024-03-17T23:50:14.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:13 smithi012 ceph-mon[31541]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:14.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:13 smithi012 ceph-mon[31541]: osdmap e27: 4 total, 4 up, 4 in 2024-03-17T23:50:15.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.157+0000 7fe21b938700 1 -- 172.21.15.79:0/3535084040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2141080b0 msgr2=0x7fe21410a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:15.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.157+0000 7fe21b938700 1 --2- 172.21.15.79:0/3535084040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2141080b0 0x7fe21410a4a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe20c00b0a0 tx=0x7fe20c00b3b0 comp rx=0 tx=0).stop 2024-03-17T23:50:15.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.158+0000 7fe21b938700 1 -- 172.21.15.79:0/3535084040 shutdown_connections 2024-03-17T23:50:15.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.158+0000 7fe21b938700 1 --2- 172.21.15.79:0/3535084040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2141080b0 0x7fe21410a4a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:15.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.158+0000 7fe21b938700 1 --2- 172.21.15.79:0/3535084040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe214102e50 0x7fe214105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:15.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.158+0000 7fe21b938700 1 --2- 172.21.15.79:0/3535084040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe214105780 0x7fe214107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:15.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.158+0000 7fe21b938700 1 -- 172.21.15.79:0/3535084040 >> 172.21.15.79:0/3535084040 conn(0x7fe2140fc7c0 msgr2=0x7fe2140febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:15.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.158+0000 7fe21b938700 1 -- 172.21.15.79:0/3535084040 shutdown_connections 2024-03-17T23:50:15.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.158+0000 7fe21b938700 1 -- 172.21.15.79:0/3535084040 wait complete. 2024-03-17T23:50:15.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.158+0000 7fe21b938700 1 Processor -- start 2024-03-17T23:50:15.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.158+0000 7fe21b938700 1 -- start start 2024-03-17T23:50:15.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.158+0000 7fe21b938700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe214102e50 0x7fe214102760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:15.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.158+0000 7fe21b938700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe214105780 0x7fe214100e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:15.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.158+0000 7fe21b938700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2141080b0 0x7fe214101340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:15.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe21b938700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe21410dfa0 con 0x7fe214105780 2024-03-17T23:50:15.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe21b938700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe21410de20 con 0x7fe2141080b0 2024-03-17T23:50:15.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe21b938700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe21410e120 con 0x7fe214102e50 2024-03-17T23:50:15.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe219ed5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2141080b0 0x7fe214101340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:15.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe219ed5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2141080b0 0x7fe214101340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52202/0 (socket says 172.21.15.79:52202) 2024-03-17T23:50:15.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe219ed5700 1 -- 172.21.15.79:0/1115637273 learned_addr learned my addr 172.21.15.79:0/1115637273 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-03-17T23:50:15.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe219ed5700 1 -- 172.21.15.79:0/1115637273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe214102e50 msgr2=0x7fe214102760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:15.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe219ed5700 1 --2- 172.21.15.79:0/1115637273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe214102e50 0x7fe214102760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:15.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe219ed5700 1 -- 172.21.15.79:0/1115637273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe214105780 msgr2=0x7fe214100e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe219ed5700 1 --2- 172.21.15.79:0/1115637273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe214105780 0x7fe214100e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe219ed5700 1 -- 172.21.15.79:0/1115637273 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe214101c10 con 0x7fe2141080b0 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe219ed5700 1 --2- 172.21.15.79:0/1115637273 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2141080b0 0x7fe214101340 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe20c007f00 tx=0x7fe20c00bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe20a7fc700 1 -- 172.21.15.79:0/1115637273 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c021030 con 0x7fe2141080b0 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe21b938700 1 -- 172.21.15.79:0/1115637273 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2141a2660 con 0x7fe2141080b0 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe20a7fc700 1 -- 172.21.15.79:0/1115637273 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe20c007bb0 con 0x7fe2141080b0 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe20a7fc700 1 -- 172.21.15.79:0/1115637273 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c01d470 con 0x7fe2141080b0 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.159+0000 7fe21b938700 1 -- 172.21.15.79:0/1115637273 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2141a2bf0 con 0x7fe2141080b0 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.160+0000 7fe21b938700 1 -- 172.21.15.79:0/1115637273 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe214068f90 con 0x7fe2141080b0 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.161+0000 7fe20a7fc700 1 -- 172.21.15.79:0/1115637273 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe20c024070 con 0x7fe2141080b0 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.162+0000 7fe20a7fc700 1 --2- 172.21.15.79:0/1115637273 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe2000a2910 0x7fe2000a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.162+0000 7fe20a7fc700 1 -- 172.21.15.79:0/1115637273 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7fe20c0cedf0 con 0x7fe2141080b0 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.162+0000 7fe2196d4700 1 --2- 172.21.15.79:0/1115637273 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe2000a2910 0x7fe2000a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:15.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.163+0000 7fe2196d4700 1 --2- 172.21.15.79:0/1115637273 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe2000a2910 0x7fe2000a4dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe2100097b0 tx=0x7fe210006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:15.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.165+0000 7fe20a7fc700 1 -- 172.21.15.79:0/1115637273 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe20c0074e0 con 0x7fe2141080b0 2024-03-17T23:50:15.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:15.368+0000 7fe21b938700 1 -- 172.21.15.79:0/1115637273 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe214062710 con 0x7fe2000a2910 2024-03-17T23:50:16.157 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:15 smithi079 ceph-mon[36241]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:16.157 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:15 smithi079 ceph-mon[36241]: osdmap e28: 4 total, 4 up, 4 in 2024-03-17T23:50:16.157 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:50:16.157 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:50:16.157 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:16.157 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:16.157 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:16.157 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:50:16.157 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:50:16.158 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:16.158 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:16.158 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:50:16.158 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:15 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:16.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:15 smithi012 ceph-mon[31541]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:16.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:15 smithi012 ceph-mon[31541]: osdmap e28: 4 total, 4 up, 4 in 2024-03-17T23:50:16.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:50:16.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:50:16.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:16.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:16.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:16.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:50:16.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:50:16.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:16.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:16.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:50:16.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:15 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:16.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:15 smithi191 ceph-mon[36708]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:16.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:15 smithi191 ceph-mon[36708]: osdmap e28: 4 total, 4 up, 4 in 2024-03-17T23:50:16.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:50:16.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:50:16.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:16.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:16.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:16.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:50:16.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:50:16.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:16.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:16.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:50:16.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:15 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:17.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:16 smithi079 ceph-mon[36241]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:50:17.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:16 smithi079 ceph-mon[36241]: Detected new or changed devices on smithi012 2024-03-17T23:50:17.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:16 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi012 to 8547M 2024-03-17T23:50:17.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:16 smithi079 ceph-mon[36241]: osdmap e29: 4 total, 4 up, 4 in 2024-03-17T23:50:17.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:16 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:17.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:16 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:17.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:16 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:50:17.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:16 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:17.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:16 smithi012 ceph-mon[31541]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:50:17.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:16 smithi012 ceph-mon[31541]: Detected new or changed devices on smithi012 2024-03-17T23:50:17.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:16 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi012 to 8547M 2024-03-17T23:50:17.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:16 smithi012 ceph-mon[31541]: osdmap e29: 4 total, 4 up, 4 in 2024-03-17T23:50:17.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:16 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:17.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:16 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:17.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:16 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:50:17.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:16 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:17.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:16 smithi191 ceph-mon[36708]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:50:17.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:16 smithi191 ceph-mon[36708]: Detected new or changed devices on smithi012 2024-03-17T23:50:17.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:16 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi012 to 8547M 2024-03-17T23:50:17.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:16 smithi191 ceph-mon[36708]: osdmap e29: 4 total, 4 up, 4 in 2024-03-17T23:50:17.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:16 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:17.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:16 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:17.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:16 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:50:17.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:16 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:18.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:17 smithi012 ceph-mon[31541]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-17T23:50:18.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:17 smithi079 ceph-mon[36241]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-17T23:50:18.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:17 smithi191 ceph-mon[36708]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-17T23:50:19.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:18 smithi012 ceph-mon[31541]: from='client.? 172.21.15.79:0/4037148016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a"}]: dispatch 2024-03-17T23:50:19.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:18 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a"}]: dispatch 2024-03-17T23:50:19.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:18 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a"}]': finished 2024-03-17T23:50:19.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:18 smithi012 ceph-mon[31541]: osdmap e30: 5 total, 4 up, 5 in 2024-03-17T23:50:19.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:18 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:19.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:18 smithi079 ceph-mon[36241]: from='client.? 172.21.15.79:0/4037148016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a"}]: dispatch 2024-03-17T23:50:19.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:18 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a"}]: dispatch 2024-03-17T23:50:19.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:18 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a"}]': finished 2024-03-17T23:50:19.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:18 smithi079 ceph-mon[36241]: osdmap e30: 5 total, 4 up, 5 in 2024-03-17T23:50:19.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:18 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:19.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:18 smithi191 ceph-mon[36708]: from='client.? 172.21.15.79:0/4037148016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a"}]: dispatch 2024-03-17T23:50:19.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:18 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a"}]: dispatch 2024-03-17T23:50:19.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:18 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a"}]': finished 2024-03-17T23:50:19.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:18 smithi191 ceph-mon[36708]: osdmap e30: 5 total, 4 up, 5 in 2024-03-17T23:50:19.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:18 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:20.160 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:19 smithi191 ceph-mon[36708]: pgmap v121: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-17T23:50:20.160 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:19 smithi191 ceph-mon[36708]: from='client.? 172.21.15.79:0/1207887282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:50:20.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:19 smithi012 ceph-mon[31541]: pgmap v121: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-17T23:50:20.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:19 smithi012 ceph-mon[31541]: from='client.? 172.21.15.79:0/1207887282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:50:20.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:19 smithi079 ceph-mon[36241]: pgmap v121: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-17T23:50:20.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:19 smithi079 ceph-mon[36241]: from='client.? 172.21.15.79:0/1207887282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:50:22.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:21 smithi012 ceph-mon[31541]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-17T23:50:22.220 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:21 smithi079 ceph-mon[36241]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-17T23:50:22.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:21 smithi191 ceph-mon[36708]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-17T23:50:24.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:23 smithi012 ceph-mon[31541]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:24.183 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:23 smithi079 ceph-mon[36241]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:24.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:23 smithi191 ceph-mon[36708]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:26.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:25 smithi079 ceph-mon[36241]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:26.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-17T23:50:26.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:26.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:25 smithi012 ceph-mon[31541]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:26.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-17T23:50:26.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:26.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:25 smithi191 ceph-mon[36708]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:26.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-17T23:50:26.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:27.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:26 smithi079 ceph-mon[36241]: Deploying daemon osd.4 on smithi079 2024-03-17T23:50:27.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:26 smithi012 ceph-mon[31541]: Deploying daemon osd.4 on smithi079 2024-03-17T23:50:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:26 smithi191 ceph-mon[36708]: Deploying daemon osd.4 on smithi079 2024-03-17T23:50:28.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:27 smithi012 ceph-mon[31541]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:28.170 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:27 smithi079 ceph-mon[36241]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:28.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:27 smithi191 ceph-mon[36708]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:30.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:29 smithi012 ceph-mon[31541]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:30.221 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:29 smithi079 ceph-mon[36241]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:30.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:29 smithi191 ceph-mon[36708]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:32.148 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:31 smithi079 ceph-mon[36241]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:32.148 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:32.149 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:32.149 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:31 smithi012 ceph-mon[31541]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:32.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:31 smithi191 ceph-mon[36708]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:32.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:32.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:32.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:34.122 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:33 smithi079 ceph-mon[36241]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:34.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:33 smithi012 ceph-mon[31541]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:34.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:33 smithi191 ceph-mon[36708]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:35.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:35.257+0000 7fe20a7fc700 1 -- 172.21.15.79:0/1115637273 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe214062710 con 0x7fe2000a2910 2024-03-17T23:50:35.259 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 4 on host 'smithi079' 2024-03-17T23:50:35.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:35.259+0000 7fe21b938700 1 -- 172.21.15.79:0/1115637273 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe2000a2910 msgr2=0x7fe2000a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:35.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:35.259+0000 7fe21b938700 1 --2- 172.21.15.79:0/1115637273 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe2000a2910 0x7fe2000a4dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe2100097b0 tx=0x7fe210006d20 comp rx=0 tx=0).stop 2024-03-17T23:50:35.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:35.259+0000 7fe21b938700 1 -- 172.21.15.79:0/1115637273 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2141080b0 msgr2=0x7fe214101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:35.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:35.259+0000 7fe21b938700 1 --2- 172.21.15.79:0/1115637273 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2141080b0 0x7fe214101340 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe20c007f00 tx=0x7fe20c00bf10 comp rx=0 tx=0).stop 2024-03-17T23:50:35.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:35.259+0000 7fe21b938700 1 -- 172.21.15.79:0/1115637273 shutdown_connections 2024-03-17T23:50:35.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:35.259+0000 7fe21b938700 1 --2- 172.21.15.79:0/1115637273 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe2000a2910 0x7fe2000a4dd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:35.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:35.259+0000 7fe21b938700 1 --2- 172.21.15.79:0/1115637273 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2141080b0 0x7fe214101340 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:35.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:35.259+0000 7fe21b938700 1 --2- 172.21.15.79:0/1115637273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe214102e50 0x7fe214102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:35.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:35.259+0000 7fe21b938700 1 --2- 172.21.15.79:0/1115637273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe214105780 0x7fe214100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:35.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:35.259+0000 7fe21b938700 1 -- 172.21.15.79:0/1115637273 >> 172.21.15.79:0/1115637273 conn(0x7fe2140fc7c0 msgr2=0x7fe214103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:35.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:35.259+0000 7fe21b938700 1 -- 172.21.15.79:0/1115637273 shutdown_connections 2024-03-17T23:50:35.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:50:35.260+0000 7fe21b938700 1 -- 172.21.15.79:0/1115637273 wait complete. 2024-03-17T23:50:35.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:35 smithi079 ceph-mon[36241]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:35.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:35.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:50:35.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:35.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:35 smithi191 ceph-mon[36708]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:35.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:35.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:50:35.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.593 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.593 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.593 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:35 smithi012 ceph-mon[31541]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:35.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:50:35.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:35.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:35.905 DEBUG:teuthology.orchestra.run.smithi079:osd.4> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.4.service 2024-03-17T23:50:35.908 INFO:tasks.cephadm:Deploying osd.5 on smithi191 with /dev/vg_nvme/lv_3... 2024-03-17T23:50:35.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-17T23:50:36.157 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-17T23:50:36.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:36 smithi079 ceph-mon[36241]: from='osd.4 [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-17T23:50:36.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:36 smithi079 ceph-mon[36241]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-17T23:50:36.590 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:36 smithi191 ceph-mon[36708]: from='osd.4 [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-17T23:50:36.590 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:36 smithi191 ceph-mon[36708]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-17T23:50:36.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:36 smithi012 ceph-mon[31541]: from='osd.4 [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-17T23:50:36.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:36 smithi012 ceph-mon[31541]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-17T23:50:37.382 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:37 smithi079 ceph-mon[36241]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:37.382 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:37 smithi079 ceph-mon[36241]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-17T23:50:37.382 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:37 smithi079 ceph-mon[36241]: from='osd.4 [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:50:37.382 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:37 smithi079 ceph-mon[36241]: osdmap e31: 5 total, 4 up, 5 in 2024-03-17T23:50:37.382 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:37 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:37.383 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:37 smithi079 ceph-mon[36241]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:50:37.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:37 smithi191 ceph-mon[36708]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:37.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:37 smithi191 ceph-mon[36708]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-17T23:50:37.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:37 smithi191 ceph-mon[36708]: from='osd.4 [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:50:37.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:37 smithi191 ceph-mon[36708]: osdmap e31: 5 total, 4 up, 5 in 2024-03-17T23:50:37.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:37 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:37.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:37 smithi191 ceph-mon[36708]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:50:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:37 smithi012 ceph-mon[31541]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:37 smithi012 ceph-mon[31541]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-17T23:50:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:37 smithi012 ceph-mon[31541]: from='osd.4 [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:50:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:37 smithi012 ceph-mon[31541]: osdmap e31: 5 total, 4 up, 5 in 2024-03-17T23:50:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:37 smithi012 ceph-mon[31541]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:50:38.046 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 17 23:50:37 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4[47849]: 2024-03-17T23:50:37.773+0000 7f3660333700 -1 osd.4 0 waiting for initial osdmap 2024-03-17T23:50:38.046 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 17 23:50:37 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4[47849]: 2024-03-17T23:50:37.779+0000 7f365811e700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:50:38.150 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-17T23:50:38.150 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-17T23:50:38.150 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-03-17T23:50:38.150 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-03-17T23:50:38.150 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114522 s, 916 MB/s 2024-03-17T23:50:38.150 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-03-17T23:50:38.451 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:38 smithi191 ceph-mon[36708]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-17T23:50:38.451 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:38 smithi191 ceph-mon[36708]: osdmap e32: 5 total, 4 up, 5 in 2024-03-17T23:50:38.451 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:38 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:38.451 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:38 smithi191 ceph-mon[36708]: from='osd.4 ' entity='osd.4' 2024-03-17T23:50:38.451 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:38 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:38.501 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:38 smithi079 ceph-mon[36241]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-17T23:50:38.501 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:38 smithi079 ceph-mon[36241]: osdmap e32: 5 total, 4 up, 5 in 2024-03-17T23:50:38.501 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:38 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:38.501 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:38 smithi079 ceph-mon[36241]: from='osd.4 ' entity='osd.4' 2024-03-17T23:50:38.501 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:38 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:38.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:38 smithi012 ceph-mon[31541]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-17T23:50:38.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:38 smithi012 ceph-mon[31541]: osdmap e32: 5 total, 4 up, 5 in 2024-03-17T23:50:38.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:38 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:38.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:38 smithi012 ceph-mon[31541]: from='osd.4 ' entity='osd.4' 2024-03-17T23:50:38.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:38 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:38.814 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi191:vg_nvme/lv_3 2024-03-17T23:50:39.057 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-17T23:50:39.428 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: purged_snaps scrub starts 2024-03-17T23:50:39.428 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: purged_snaps scrub ok 2024-03-17T23:50:39.428 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:39.428 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: osd.4 [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360] boot 2024-03-17T23:50:39.429 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: osdmap e33: 5 total, 5 up, 5 in 2024-03-17T23:50:39.429 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:39.429 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: Detected new or changed devices on smithi079 2024-03-17T23:50:39.429 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:39.429 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:39.429 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:50:39.429 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:50:39.429 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi079 to 8540M 2024-03-17T23:50:39.429 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:39.429 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:39.429 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:50:39.429 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:39.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: purged_snaps scrub starts 2024-03-17T23:50:39.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: purged_snaps scrub ok 2024-03-17T23:50:39.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:39.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: osd.4 [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360] boot 2024-03-17T23:50:39.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: osdmap e33: 5 total, 5 up, 5 in 2024-03-17T23:50:39.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:39.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: Detected new or changed devices on smithi079 2024-03-17T23:50:39.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:39.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:39.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:50:39.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:50:39.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi079 to 8540M 2024-03-17T23:50:39.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:39.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:39.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:50:39.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:39.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: purged_snaps scrub starts 2024-03-17T23:50:39.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: purged_snaps scrub ok 2024-03-17T23:50:39.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-17T23:50:39.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: osd.4 [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360] boot 2024-03-17T23:50:39.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: osdmap e33: 5 total, 5 up, 5 in 2024-03-17T23:50:39.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-17T23:50:39.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: Detected new or changed devices on smithi079 2024-03-17T23:50:39.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:39.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:39.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:50:39.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:50:39.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi079 to 8540M 2024-03-17T23:50:39.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:39.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:39.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:50:39.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:40.630 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.625+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/2596334964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b14103500 msgr2=0x7f1b14103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:40.630 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.625+0000 7f1b1bf9e700 1 --2- 172.21.15.191:0/2596334964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b14103500 0x7f1b14103960 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1b04009990 tx=0x7f1b04009ca0 comp rx=0 tx=0).stop 2024-03-17T23:50:40.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.625+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/2596334964 shutdown_connections 2024-03-17T23:50:40.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.625+0000 7f1b1bf9e700 1 --2- 172.21.15.191:0/2596334964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b14103ea0 0x7f1b1410a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:40.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.625+0000 7f1b1bf9e700 1 --2- 172.21.15.191:0/2596334964 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b14102300 0x7f1b14102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:40.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.625+0000 7f1b1bf9e700 1 --2- 172.21.15.191:0/2596334964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b14103500 0x7f1b14103960 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:40.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.625+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/2596334964 >> 172.21.15.191:0/2596334964 conn(0x7f1b140fd8e0 msgr2=0x7f1b140ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:40.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.625+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/2596334964 shutdown_connections 2024-03-17T23:50:40.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.625+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/2596334964 wait complete. 2024-03-17T23:50:40.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.625+0000 7f1b1bf9e700 1 Processor -- start 2024-03-17T23:50:40.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.625+0000 7f1b1bf9e700 1 -- start start 2024-03-17T23:50:40.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1bf9e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b14102300 0x7f1b14076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:40.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1bf9e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b14103500 0x7f1b140746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:40.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1bf9e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b14103ea0 0x7f1b14074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:40.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1bf9e700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1410dfb0 con 0x7f1b14102300 2024-03-17T23:50:40.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1bf9e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1410de30 con 0x7f1b14103500 2024-03-17T23:50:40.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1bf9e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1410e130 con 0x7f1b14103ea0 2024-03-17T23:50:40.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1a53b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b14103ea0 0x7f1b14074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:40.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1a53b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b14103ea0 0x7f1b14074c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:51306/0 (socket says 172.21.15.191:51306) 2024-03-17T23:50:40.634 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1a53b700 1 -- 172.21.15.191:0/3099670976 learned_addr learned my addr 172.21.15.191:0/3099670976 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-17T23:50:40.634 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b19539700 1 --2- 172.21.15.191:0/3099670976 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b14103500 0x7f1b140746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:40.634 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1a53b700 1 -- 172.21.15.191:0/3099670976 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b14103500 msgr2=0x7f1b140746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:40.634 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1a53b700 1 --2- 172.21.15.191:0/3099670976 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b14103500 0x7f1b140746c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:40.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1a53b700 1 -- 172.21.15.191:0/3099670976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b14102300 msgr2=0x7f1b14076070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:40.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1a53b700 1 --2- 172.21.15.191:0/3099670976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b14102300 0x7f1b14076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:40.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1a53b700 1 -- 172.21.15.191:0/3099670976 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b140754b0 con 0x7f1b14103ea0 2024-03-17T23:50:40.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b1a53b700 1 --2- 172.21.15.191:0/3099670976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b14103ea0 0x7f1b14074c00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00b970 tx=0x7f1b0c00bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:40.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b0affd700 1 -- 172.21.15.191:0/3099670976 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c019070 con 0x7f1b14103ea0 2024-03-17T23:50:40.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b0affd700 1 -- 172.21.15.191:0/3099670976 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b0c014440 con 0x7f1b14103ea0 2024-03-17T23:50:40.636 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.626+0000 7f1b0affd700 1 -- 172.21.15.191:0/3099670976 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c0075d0 con 0x7f1b14103ea0 2024-03-17T23:50:40.636 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.627+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/3099670976 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b141a2780 con 0x7f1b14103ea0 2024-03-17T23:50:40.636 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.627+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/3099670976 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b141a2cc0 con 0x7f1b14103ea0 2024-03-17T23:50:40.636 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.627+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/3099670976 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b14068f90 con 0x7f1b14103ea0 2024-03-17T23:50:40.636 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.629+0000 7f1b0affd700 1 -- 172.21.15.191:0/3099670976 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1b0c01c070 con 0x7f1b14103ea0 2024-03-17T23:50:40.636 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.629+0000 7f1b0affd700 1 --2- 172.21.15.191:0/3099670976 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1b000a2890 0x7f1b000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:40.637 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.629+0000 7f1b0affd700 1 -- 172.21.15.191:0/3099670976 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f1b0c0cd310 con 0x7f1b14103ea0 2024-03-17T23:50:40.637 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.630+0000 7f1b19d3a700 1 --2- 172.21.15.191:0/3099670976 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1b000a2890 0x7f1b000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:40.638 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.631+0000 7f1b19d3a700 1 --2- 172.21.15.191:0/3099670976 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1b000a2890 0x7f1b000a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b14103360 tx=0x7f1b10009470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:40.638 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.633+0000 7f1b0affd700 1 -- 172.21.15.191:0/3099670976 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1b0c09b420 con 0x7f1b14103ea0 2024-03-17T23:50:40.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:40.836+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/3099670976 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1b14062710 con 0x7f1b000a2890 2024-03-17T23:50:41.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:41 smithi191 ceph-mon[36708]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:41.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:41 smithi191 ceph-mon[36708]: osdmap e34: 5 total, 5 up, 5 in 2024-03-17T23:50:41.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:41 smithi191 ceph-mon[36708]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:50:41.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:41 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:50:41.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:41 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:50:41.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:41 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:41.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:41 smithi012 ceph-mon[31541]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:41.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:41 smithi012 ceph-mon[31541]: osdmap e34: 5 total, 5 up, 5 in 2024-03-17T23:50:41.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:41 smithi012 ceph-mon[31541]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:50:41.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:41 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:50:41.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:41 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:50:41.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:41 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:41.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:41 smithi079 ceph-mon[36241]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:41.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:41 smithi079 ceph-mon[36241]: osdmap e34: 5 total, 5 up, 5 in 2024-03-17T23:50:41.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:41 smithi079 ceph-mon[36241]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:50:41.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:41 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:50:41.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:41 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:50:41.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:41 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:43.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:43 smithi191 ceph-mon[36708]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:43.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:43 smithi012 ceph-mon[31541]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:43.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:43 smithi079 ceph-mon[36241]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:44.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:44 smithi191 ceph-mon[36708]: from='client.? 172.21.15.191:0/4068794037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3a56fd4-99df-4230-a51f-382af53afef2"}]: dispatch 2024-03-17T23:50:44.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:44 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3a56fd4-99df-4230-a51f-382af53afef2"}]: dispatch 2024-03-17T23:50:44.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:44 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3a56fd4-99df-4230-a51f-382af53afef2"}]': finished 2024-03-17T23:50:44.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:44 smithi191 ceph-mon[36708]: osdmap e35: 6 total, 5 up, 6 in 2024-03-17T23:50:44.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-17T23:50:44.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:44 smithi191 ceph-mon[36708]: from='client.? 172.21.15.191:0/3384472015' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:50:44.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:44 smithi012 ceph-mon[31541]: from='client.? 172.21.15.191:0/4068794037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3a56fd4-99df-4230-a51f-382af53afef2"}]: dispatch 2024-03-17T23:50:44.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:44 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3a56fd4-99df-4230-a51f-382af53afef2"}]: dispatch 2024-03-17T23:50:44.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:44 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3a56fd4-99df-4230-a51f-382af53afef2"}]': finished 2024-03-17T23:50:44.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:44 smithi012 ceph-mon[31541]: osdmap e35: 6 total, 5 up, 6 in 2024-03-17T23:50:44.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-17T23:50:44.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:44 smithi012 ceph-mon[31541]: from='client.? 172.21.15.191:0/3384472015' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:50:44.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:44 smithi079 ceph-mon[36241]: from='client.? 172.21.15.191:0/4068794037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3a56fd4-99df-4230-a51f-382af53afef2"}]: dispatch 2024-03-17T23:50:44.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:44 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3a56fd4-99df-4230-a51f-382af53afef2"}]: dispatch 2024-03-17T23:50:44.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:44 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3a56fd4-99df-4230-a51f-382af53afef2"}]': finished 2024-03-17T23:50:44.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:44 smithi079 ceph-mon[36241]: osdmap e35: 6 total, 5 up, 6 in 2024-03-17T23:50:44.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-17T23:50:44.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:44 smithi079 ceph-mon[36241]: from='client.? 172.21.15.191:0/3384472015' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:50:45.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:45 smithi191 ceph-mon[36708]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:45.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:45 smithi012 ceph-mon[31541]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:45.969 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:45 smithi079 ceph-mon[36241]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:47.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:47 smithi191 ceph-mon[36708]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:47.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:47 smithi012 ceph-mon[31541]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:48.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:47 smithi079 ceph-mon[36241]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:49.796 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:49 smithi191 ceph-mon[36708]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:49.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:49 smithi012 ceph-mon[31541]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:50.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:49 smithi079 ceph-mon[36241]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:51.820 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:51 smithi191 ceph-mon[36708]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:51.820 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-17T23:50:51.821 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:51.821 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:51 smithi191 ceph-mon[36708]: Deploying daemon osd.5 on smithi191 2024-03-17T23:50:51.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:51 smithi012 ceph-mon[31541]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:51.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-17T23:50:51.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:51.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:51 smithi012 ceph-mon[31541]: Deploying daemon osd.5 on smithi191 2024-03-17T23:50:52.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:51 smithi079 ceph-mon[36241]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:52.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-17T23:50:52.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:50:52.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:51 smithi079 ceph-mon[36241]: Deploying daemon osd.5 on smithi191 2024-03-17T23:50:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:53 smithi191 ceph-mon[36708]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:53.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:53 smithi012 ceph-mon[31541]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:54.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:53 smithi079 ceph-mon[36241]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:55.683 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:55 smithi191 ceph-mon[36708]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:55.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:55 smithi012 ceph-mon[31541]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:56.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:55 smithi079 ceph-mon[36241]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:57 smithi191 ceph-mon[36708]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:57.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:57.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:57 smithi012 ceph-mon[31541]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:57.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:57.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:57.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:58.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:57 smithi079 ceph-mon[36241]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:58.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:58.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:50:58.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:50:59.727 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:50:59 smithi191 ceph-mon[36708]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:50:59.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:59.824+0000 7f1b0affd700 1 -- 172.21.15.191:0/3099670976 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1b14062710 con 0x7f1b000a2890 2024-03-17T23:50:59.825 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 5 on host 'smithi191' 2024-03-17T23:50:59.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:59.826+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/3099670976 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1b000a2890 msgr2=0x7f1b000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:59.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:59.826+0000 7f1b1bf9e700 1 --2- 172.21.15.191:0/3099670976 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1b000a2890 0x7f1b000a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b14103360 tx=0x7f1b10009470 comp rx=0 tx=0).stop 2024-03-17T23:50:59.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:59.826+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/3099670976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b14103ea0 msgr2=0x7f1b14074c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:59.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:59.826+0000 7f1b1bf9e700 1 --2- 172.21.15.191:0/3099670976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b14103ea0 0x7f1b14074c00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00b970 tx=0x7f1b0c00bd30 comp rx=0 tx=0).stop 2024-03-17T23:50:59.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:59.826+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/3099670976 shutdown_connections 2024-03-17T23:50:59.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:59.826+0000 7f1b1bf9e700 1 --2- 172.21.15.191:0/3099670976 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1b000a2890 0x7f1b000a4d50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:59.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:59.826+0000 7f1b1bf9e700 1 --2- 172.21.15.191:0/3099670976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b14103ea0 0x7f1b14074c00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:59.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:59.826+0000 7f1b1bf9e700 1 --2- 172.21.15.191:0/3099670976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b14102300 0x7f1b14076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:59.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:59.826+0000 7f1b1bf9e700 1 --2- 172.21.15.191:0/3099670976 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b14103500 0x7f1b140746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:59.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:59.826+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/3099670976 >> 172.21.15.191:0/3099670976 conn(0x7f1b140fd8e0 msgr2=0x7f1b14106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:59.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:59.826+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/3099670976 shutdown_connections 2024-03-17T23:50:59.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:50:59.827+0000 7f1b1bf9e700 1 -- 172.21.15.191:0/3099670976 wait complete. 2024-03-17T23:50:59.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:50:59 smithi012 ceph-mon[31541]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:51:00.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:50:59 smithi079 ceph-mon[36241]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:51:00.785 DEBUG:teuthology.orchestra.run.smithi191:osd.5> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.5.service 2024-03-17T23:51:00.788 INFO:tasks.cephadm:Deploying osd.6 on smithi012 with /dev/vg_nvme/lv_2... 2024-03-17T23:51:00.788 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-17T23:51:01.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:00 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:01.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:00 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:01.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:00 smithi079 ceph-mon[36241]: from='osd.5 [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-17T23:51:01.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:00 smithi079 ceph-mon[36241]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-17T23:51:01.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:00 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:01.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:00 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:01.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:00 smithi191 ceph-mon[36708]: from='osd.5 [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-17T23:51:01.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:00 smithi191 ceph-mon[36708]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-17T23:51:01.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:00 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:01.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:00 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:01.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:00 smithi012 ceph-mon[31541]: from='osd.5 [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-17T23:51:01.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:00 smithi012 ceph-mon[31541]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-17T23:51:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:51:02.016 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:01 smithi012 ceph-mon[31541]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:51:02.016 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:01 smithi012 ceph-mon[31541]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-17T23:51:02.016 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:01 smithi012 ceph-mon[31541]: osdmap e36: 6 total, 5 up, 6 in 2024-03-17T23:51:02.016 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:01 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-17T23:51:02.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:01 smithi012 ceph-mon[31541]: from='osd.5 [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:51:02.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:01 smithi012 ceph-mon[31541]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:51:02.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:01 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:02.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:01 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:02.094 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 17 23:51:01 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-17T23:51:01.811+0000 7fcbf4049700 -1 osd.5 0 waiting for initial osdmap 2024-03-17T23:51:02.094 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 17 23:51:01 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-17T23:51:01.817+0000 7fcbefe3c700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:51:02.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:01 smithi191 ceph-mon[36708]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:51:02.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:01 smithi191 ceph-mon[36708]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-17T23:51:02.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:01 smithi191 ceph-mon[36708]: osdmap e36: 6 total, 5 up, 6 in 2024-03-17T23:51:02.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:01 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-17T23:51:02.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:01 smithi191 ceph-mon[36708]: from='osd.5 [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:51:02.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:01 smithi191 ceph-mon[36708]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:51:02.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:01 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:02.095 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:01 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:02.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:01 smithi079 ceph-mon[36241]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:51:02.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:01 smithi079 ceph-mon[36241]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-17T23:51:02.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:01 smithi079 ceph-mon[36241]: osdmap e36: 6 total, 5 up, 6 in 2024-03-17T23:51:02.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:01 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-17T23:51:02.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:01 smithi079 ceph-mon[36241]: from='osd.5 [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:51:02.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:01 smithi079 ceph-mon[36241]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:51:02.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:01 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:02.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:01 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:02.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:02 smithi191 ceph-mon[36708]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-17T23:51:02.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:02 smithi191 ceph-mon[36708]: osdmap e37: 6 total, 5 up, 6 in 2024-03-17T23:51:02.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:02 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-17T23:51:03.137 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:02 smithi012 ceph-mon[31541]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-17T23:51:03.137 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:02 smithi012 ceph-mon[31541]: osdmap e37: 6 total, 5 up, 6 in 2024-03-17T23:51:03.137 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:02 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-17T23:51:03.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:02 smithi079 ceph-mon[36241]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-17T23:51:03.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:02 smithi079 ceph-mon[36241]: osdmap e37: 6 total, 5 up, 6 in 2024-03-17T23:51:03.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:02 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-17T23:51:04.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:03 smithi191 ceph-mon[36708]: purged_snaps scrub starts 2024-03-17T23:51:04.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:03 smithi191 ceph-mon[36708]: purged_snaps scrub ok 2024-03-17T23:51:04.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:03 smithi191 ceph-mon[36708]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:51:04.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:03 smithi191 ceph-mon[36708]: osd.5 [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194] boot 2024-03-17T23:51:04.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:03 smithi191 ceph-mon[36708]: osdmap e38: 6 total, 6 up, 6 in 2024-03-17T23:51:04.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-17T23:51:04.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:03 smithi191 ceph-mon[36708]: osdmap e39: 6 total, 6 up, 6 in 2024-03-17T23:51:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:03 smithi012 ceph-mon[31541]: purged_snaps scrub starts 2024-03-17T23:51:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:03 smithi012 ceph-mon[31541]: purged_snaps scrub ok 2024-03-17T23:51:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:03 smithi012 ceph-mon[31541]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:51:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:03 smithi012 ceph-mon[31541]: osd.5 [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194] boot 2024-03-17T23:51:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:03 smithi012 ceph-mon[31541]: osdmap e38: 6 total, 6 up, 6 in 2024-03-17T23:51:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-17T23:51:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:03 smithi012 ceph-mon[31541]: osdmap e39: 6 total, 6 up, 6 in 2024-03-17T23:51:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:03 smithi079 ceph-mon[36241]: purged_snaps scrub starts 2024-03-17T23:51:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:03 smithi079 ceph-mon[36241]: purged_snaps scrub ok 2024-03-17T23:51:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:03 smithi079 ceph-mon[36241]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-17T23:51:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:03 smithi079 ceph-mon[36241]: osd.5 [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194] boot 2024-03-17T23:51:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:03 smithi079 ceph-mon[36241]: osdmap e38: 6 total, 6 up, 6 in 2024-03-17T23:51:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:03 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-17T23:51:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:03 smithi079 ceph-mon[36241]: osdmap e39: 6 total, 6 up, 6 in 2024-03-17T23:51:04.414 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-17T23:51:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-17T23:51:04.415 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-03-17T23:51:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-03-17T23:51:04.415 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0336866 s, 311 MB/s 2024-03-17T23:51:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-03-17T23:51:05.371 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi012:vg_nvme/lv_2 2024-03-17T23:51:05.460 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:05.460 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: osdmap e40: 6 total, 6 up, 6 in 2024-03-17T23:51:05.461 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: Detected new or changed devices on smithi191 2024-03-17T23:51:05.461 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.461 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.461 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:05.461 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:05.461 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi191 to 8540M 2024-03-17T23:51:05.461 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.461 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:05.461 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:51:05.461 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.461 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:51:05.461 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:05.462 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:51:05.462 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:05.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: osdmap e40: 6 total, 6 up, 6 in 2024-03-17T23:51:05.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: Detected new or changed devices on smithi191 2024-03-17T23:51:05.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:05.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:05.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi191 to 8540M 2024-03-17T23:51:05.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:05.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:51:05.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:51:05.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:05.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:51:05.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:05.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: osdmap e40: 6 total, 6 up, 6 in 2024-03-17T23:51:05.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: Detected new or changed devices on smithi191 2024-03-17T23:51:05.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:05.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:05.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi191 to 8540M 2024-03-17T23:51:05.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:05.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:51:05.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:51:05.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:05.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:51:05.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:05.838 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:51:06.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:06 smithi012 ceph-mon[31541]: osdmap e41: 6 total, 6 up, 6 in 2024-03-17T23:51:06.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:06 smithi012 ceph-mon[31541]: osdmap e42: 6 total, 6 up, 6 in 2024-03-17T23:51:06.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:06 smithi079 ceph-mon[36241]: osdmap e41: 6 total, 6 up, 6 in 2024-03-17T23:51:06.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:06 smithi079 ceph-mon[36241]: osdmap e42: 6 total, 6 up, 6 in 2024-03-17T23:51:06.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:06 smithi191 ceph-mon[36708]: osdmap e41: 6 total, 6 up, 6 in 2024-03-17T23:51:06.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:06 smithi191 ceph-mon[36708]: osdmap e42: 6 total, 6 up, 6 in 2024-03-17T23:51:07.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:07 smithi012 ceph-mon[31541]: pgmap v157: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-17T23:51:07.742 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:07 smithi191 ceph-mon[36708]: pgmap v157: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-17T23:51:07.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:07 smithi079 ceph-mon[36241]: pgmap v157: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-17T23:51:07.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.942+0000 7fa342000700 1 -- 172.21.15.12:0/669316781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa33c105760 msgr2=0x7fa33c107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:51:07.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.942+0000 7fa342000700 1 --2- 172.21.15.12:0/669316781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa33c105760 0x7fa33c107b50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa3280098f0 tx=0x7fa328009c00 comp rx=0 tx=0).stop 2024-03-17T23:51:07.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.942+0000 7fa342000700 1 -- 172.21.15.12:0/669316781 shutdown_connections 2024-03-17T23:51:07.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.942+0000 7fa342000700 1 --2- 172.21.15.12:0/669316781 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa33c108090 0x7fa33c10a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:07.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.942+0000 7fa342000700 1 --2- 172.21.15.12:0/669316781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa33c102e30 0x7fa33c105220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:07.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.942+0000 7fa342000700 1 --2- 172.21.15.12:0/669316781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa33c105760 0x7fa33c107b50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:07.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.942+0000 7fa342000700 1 -- 172.21.15.12:0/669316781 >> 172.21.15.12:0/669316781 conn(0x7fa33c0fc7a0 msgr2=0x7fa33c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:51:07.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.942+0000 7fa342000700 1 -- 172.21.15.12:0/669316781 shutdown_connections 2024-03-17T23:51:07.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.942+0000 7fa342000700 1 -- 172.21.15.12:0/669316781 wait complete. 2024-03-17T23:51:07.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.943+0000 7fa342000700 1 Processor -- start 2024-03-17T23:51:07.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.944+0000 7fa342000700 1 -- start start 2024-03-17T23:51:07.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.944+0000 7fa342000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa33c102e30 0x7fa33c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:51:07.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.944+0000 7fa342000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa33c105760 0x7fa33c19ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:51:07.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.944+0000 7fa342000700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa33c108090 0x7fa33c1a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:51:07.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.944+0000 7fa342000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa33c10ded0 con 0x7fa33c102e30 2024-03-17T23:51:07.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.944+0000 7fa342000700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa33c10dd50 con 0x7fa33c108090 2024-03-17T23:51:07.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.944+0000 7fa342000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa33c10e050 con 0x7fa33c105760 2024-03-17T23:51:07.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.944+0000 7fa33b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa33c102e30 0x7fa33c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:51:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.944+0000 7fa33b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa33c102e30 0x7fa33c19a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33668/0 (socket says 172.21.15.12:33668) 2024-03-17T23:51:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.944+0000 7fa33b400700 1 -- 172.21.15.12:0/386591911 learned_addr learned my addr 172.21.15.12:0/386591911 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:51:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.944+0000 7fa33be00700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa33c108090 0x7fa33c1a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:51:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.944+0000 7fa33aa00700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa33c105760 0x7fa33c19ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:51:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.945+0000 7fa33b400700 1 -- 172.21.15.12:0/386591911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa33c105760 msgr2=0x7fa33c19ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:51:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.945+0000 7fa33b400700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa33c105760 0x7fa33c19ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.945+0000 7fa33b400700 1 -- 172.21.15.12:0/386591911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa33c108090 msgr2=0x7fa33c1a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:51:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.945+0000 7fa33b400700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa33c108090 0x7fa33c1a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.945+0000 7fa33b400700 1 -- 172.21.15.12:0/386591911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33c1a2400 con 0x7fa33c102e30 2024-03-17T23:51:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.945+0000 7fa33be00700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa33c108090 0x7fa33c1a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:51:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.945+0000 7fa33b400700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa33c102e30 0x7fa33c19a7a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00e9f0 tx=0x7fa32c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:51:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.945+0000 7fa333e00700 1 -- 172.21.15.12:0/386591911 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c004a30 con 0x7fa33c102e30 2024-03-17T23:51:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.946+0000 7fa333e00700 1 -- 172.21.15.12:0/386591911 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa32c005500 con 0x7fa33c102e30 2024-03-17T23:51:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.946+0000 7fa333e00700 1 -- 172.21.15.12:0/386591911 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c00cd60 con 0x7fa33c102e30 2024-03-17T23:51:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.946+0000 7fa342000700 1 -- 172.21.15.12:0/386591911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33c1a26f0 con 0x7fa33c102e30 2024-03-17T23:51:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.946+0000 7fa342000700 1 -- 172.21.15.12:0/386591911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa33c1a2c30 con 0x7fa33c102e30 2024-03-17T23:51:07.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.947+0000 7fa342000700 1 -- 172.21.15.12:0/386591911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa33c068f90 con 0x7fa33c102e30 2024-03-17T23:51:07.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.949+0000 7fa333e00700 1 -- 172.21.15.12:0/386591911 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa32c004c60 con 0x7fa33c102e30 2024-03-17T23:51:07.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.950+0000 7fa333e00700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3200a2890 0x7fa3200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:51:07.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.951+0000 7fa333e00700 1 -- 172.21.15.12:0/386591911 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7fa32c0cdd80 con 0x7fa33c102e30 2024-03-17T23:51:07.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.951+0000 7fa33aa00700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3200a2890 0x7fa3200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:51:07.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.951+0000 7fa33aa00700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3200a2890 0x7fa3200a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa328009810 tx=0x7fa328019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:51:07.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:07.956+0000 7fa333e00700 1 -- 172.21.15.12:0/386591911 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa32c09bc40 con 0x7fa33c102e30 2024-03-17T23:51:08.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:08.194+0000 7fa342000700 1 -- 172.21.15.12:0/386591911 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa33c062710 con 0x7fa3200a2890 2024-03-17T23:51:08.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:51:08.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:51:08.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:08.756 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:51:08.756 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:51:08.756 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:08.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:51:08.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:51:08.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:09.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:09 smithi012 ceph-mon[31541]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-17T23:51:09.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:09 smithi012 ceph-mon[31541]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:51:09.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:09 smithi079 ceph-mon[36241]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-17T23:51:09.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:09 smithi079 ceph-mon[36241]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:51:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:09 smithi191 ceph-mon[36708]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-17T23:51:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:09 smithi191 ceph-mon[36708]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:51:11.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:11 smithi012 ceph-mon[31541]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-17T23:51:11.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:11 smithi079 ceph-mon[36241]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-17T23:51:11.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:11 smithi191 ceph-mon[36708]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-17T23:51:12.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:12 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3008584981' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "309e6139-7f6e-4a92-bce4-e240cdc72f94"}]: dispatch 2024-03-17T23:51:12.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:12 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3008584981' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "309e6139-7f6e-4a92-bce4-e240cdc72f94"}]': finished 2024-03-17T23:51:12.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:12 smithi012 ceph-mon[31541]: osdmap e43: 7 total, 6 up, 7 in 2024-03-17T23:51:12.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:12.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:12 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3008584981' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "309e6139-7f6e-4a92-bce4-e240cdc72f94"}]: dispatch 2024-03-17T23:51:12.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:12 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3008584981' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "309e6139-7f6e-4a92-bce4-e240cdc72f94"}]': finished 2024-03-17T23:51:12.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:12 smithi079 ceph-mon[36241]: osdmap e43: 7 total, 6 up, 7 in 2024-03-17T23:51:12.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:12.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:12 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3008584981' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "309e6139-7f6e-4a92-bce4-e240cdc72f94"}]: dispatch 2024-03-17T23:51:12.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:12 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3008584981' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "309e6139-7f6e-4a92-bce4-e240cdc72f94"}]': finished 2024-03-17T23:51:12.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:12 smithi191 ceph-mon[36708]: osdmap e43: 7 total, 6 up, 7 in 2024-03-17T23:51:12.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:13.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:13 smithi012 ceph-mon[31541]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-17T23:51:13.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:13 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/52526868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:51:13.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:13 smithi079 ceph-mon[36241]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-17T23:51:13.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:13 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/52526868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:51:13.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:13 smithi191 ceph-mon[36708]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-17T23:51:13.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:13 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/52526868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:51:15.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:15 smithi012 ceph-mon[31541]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:15.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:15 smithi079 ceph-mon[36241]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:15.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:15 smithi191 ceph-mon[36708]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:17.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:17 smithi012 ceph-mon[31541]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:17.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:17 smithi079 ceph-mon[36241]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:17.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:17 smithi191 ceph-mon[36708]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:19.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:19 smithi012 ceph-mon[31541]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:19.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:19 smithi079 ceph-mon[36241]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:19.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:19 smithi191 ceph-mon[36708]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:21.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:21 smithi012 ceph-mon[31541]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:21.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:21 smithi079 ceph-mon[36241]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:21.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:21 smithi191 ceph-mon[36708]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:22.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:22 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-17T23:51:22.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:22 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:22.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:22 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-17T23:51:22.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:22 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:22.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:22 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-17T23:51:22.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:22 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:23.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:23 smithi012 ceph-mon[31541]: Deploying daemon osd.6 on smithi012 2024-03-17T23:51:23.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:23 smithi012 ceph-mon[31541]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:23.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:23 smithi079 ceph-mon[36241]: Deploying daemon osd.6 on smithi012 2024-03-17T23:51:23.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:23 smithi079 ceph-mon[36241]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:23.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:23 smithi191 ceph-mon[36708]: Deploying daemon osd.6 on smithi012 2024-03-17T23:51:23.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:23 smithi191 ceph-mon[36708]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:25.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:25 smithi012 ceph-mon[31541]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:25.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:25 smithi079 ceph-mon[36241]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:25.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:25 smithi191 ceph-mon[36708]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:27.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:27 smithi012 ceph-mon[31541]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:27.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:27 smithi079 ceph-mon[36241]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:27.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:27 smithi191 ceph-mon[36708]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:29.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:29 smithi012 ceph-mon[31541]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:29.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:29 smithi079 ceph-mon[36241]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:29.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:29 smithi191 ceph-mon[36708]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:31 smithi012 ceph-mon[31541]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:51:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:31.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:31 smithi079 ceph-mon[36241]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:51:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:31.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:31 smithi191 ceph-mon[36708]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:31.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:51:31.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:31.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:33.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:33 smithi079 ceph-mon[36241]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:33.832 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:33 smithi012 ceph-mon[31541]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:33.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:33 smithi191 ceph-mon[36708]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:35 smithi012 ceph-mon[31541]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:35 smithi012 ceph-mon[31541]: from='osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-17T23:51:35.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:35 smithi079 ceph-mon[36241]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:35 smithi079 ceph-mon[36241]: from='osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-17T23:51:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:35 smithi191 ceph-mon[36708]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:35 smithi191 ceph-mon[36708]: from='osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-17T23:51:35.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:35.846+0000 7fa333e00700 1 -- 172.21.15.12:0/386591911 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa33c062710 con 0x7fa3200a2890 2024-03-17T23:51:35.847 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 6 on host 'smithi012' 2024-03-17T23:51:35.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:35.849+0000 7fa342000700 1 -- 172.21.15.12:0/386591911 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3200a2890 msgr2=0x7fa3200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:51:35.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:35.849+0000 7fa342000700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3200a2890 0x7fa3200a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa328009810 tx=0x7fa328019040 comp rx=0 tx=0).stop 2024-03-17T23:51:35.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:35.849+0000 7fa342000700 1 -- 172.21.15.12:0/386591911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa33c102e30 msgr2=0x7fa33c19a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:51:35.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:35.849+0000 7fa342000700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa33c102e30 0x7fa33c19a7a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00e9f0 tx=0x7fa32c00ed00 comp rx=0 tx=0).stop 2024-03-17T23:51:35.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:35.850+0000 7fa342000700 1 -- 172.21.15.12:0/386591911 shutdown_connections 2024-03-17T23:51:35.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:35.850+0000 7fa342000700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3200a2890 0x7fa3200a4d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:35.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:35.850+0000 7fa342000700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa33c108090 0x7fa33c1a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:35.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:35.850+0000 7fa342000700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa33c102e30 0x7fa33c19a7a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:35.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:35.850+0000 7fa342000700 1 --2- 172.21.15.12:0/386591911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa33c105760 0x7fa33c19ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:35.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:35.850+0000 7fa342000700 1 -- 172.21.15.12:0/386591911 >> 172.21.15.12:0/386591911 conn(0x7fa33c0fc7a0 msgr2=0x7fa33c102610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:51:35.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:35.850+0000 7fa342000700 1 -- 172.21.15.12:0/386591911 shutdown_connections 2024-03-17T23:51:35.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:51:35.850+0000 7fa342000700 1 -- 172.21.15.12:0/386591911 wait complete. 2024-03-17T23:51:36.674 DEBUG:teuthology.orchestra.run.smithi012:osd.6> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.6.service 2024-03-17T23:51:36.678 INFO:tasks.cephadm:Deploying osd.7 on smithi079 with /dev/vg_nvme/lv_2... 2024-03-17T23:51:36.678 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-17T23:51:36.708 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:36 smithi079 ceph-mon[36241]: from='osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-17T23:51:36.708 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:36 smithi079 ceph-mon[36241]: osdmap e44: 7 total, 6 up, 7 in 2024-03-17T23:51:36.708 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:36 smithi079 ceph-mon[36241]: from='osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:51:36.708 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:36 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:36.708 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:36 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:36.708 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:36 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:51:36.708 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:36 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:36.708 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:36 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:36.708 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:36 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:36.709 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:36 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:51:36.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:36 smithi012 ceph-mon[31541]: from='osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-17T23:51:36.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:36 smithi012 ceph-mon[31541]: osdmap e44: 7 total, 6 up, 7 in 2024-03-17T23:51:36.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:36 smithi012 ceph-mon[31541]: from='osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:51:36.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:36 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:36.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:36 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:36.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:36 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:51:36.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:36 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:36.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:36 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:36.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:36 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:36.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:36 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:51:36.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:36 smithi191 ceph-mon[36708]: from='osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-17T23:51:36.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:36 smithi191 ceph-mon[36708]: osdmap e44: 7 total, 6 up, 7 in 2024-03-17T23:51:36.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:36 smithi191 ceph-mon[36708]: from='osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:51:36.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:36 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:36.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:36 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:36.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:36 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:51:36.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:36 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:36.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:36 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:36.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:36 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:36.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:36 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:51:36.926 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.b/config 2024-03-17T23:51:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:37 smithi012 ceph-mon[31541]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:37 smithi012 ceph-mon[31541]: from='osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-17T23:51:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:37 smithi012 ceph-mon[31541]: osdmap e45: 7 total, 6 up, 7 in 2024-03-17T23:51:37.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:37.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:37.742 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:37 smithi191 ceph-mon[36708]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:37.742 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:37 smithi191 ceph-mon[36708]: from='osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-17T23:51:37.742 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:37 smithi191 ceph-mon[36708]: osdmap e45: 7 total, 6 up, 7 in 2024-03-17T23:51:37.742 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:37 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:37.743 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:37 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:37.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:37 smithi079 ceph-mon[36241]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-17T23:51:37.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:37 smithi079 ceph-mon[36241]: from='osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-17T23:51:37.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:37 smithi079 ceph-mon[36241]: osdmap e45: 7 total, 6 up, 7 in 2024-03-17T23:51:37.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:37 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:37.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:37 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:38.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:38 smithi079 ceph-mon[36241]: purged_snaps scrub starts 2024-03-17T23:51:38.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:38 smithi079 ceph-mon[36241]: purged_snaps scrub ok 2024-03-17T23:51:38.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:38 smithi079 ceph-mon[36241]: osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073] boot 2024-03-17T23:51:38.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:38 smithi079 ceph-mon[36241]: osdmap e46: 7 total, 7 up, 7 in 2024-03-17T23:51:38.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:38 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:38.667 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:38 smithi012 ceph-mon[31541]: purged_snaps scrub starts 2024-03-17T23:51:38.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:38 smithi012 ceph-mon[31541]: purged_snaps scrub ok 2024-03-17T23:51:38.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:38 smithi012 ceph-mon[31541]: osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073] boot 2024-03-17T23:51:38.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:38 smithi012 ceph-mon[31541]: osdmap e46: 7 total, 7 up, 7 in 2024-03-17T23:51:38.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:38 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:38.755 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:38 smithi191 ceph-mon[36708]: purged_snaps scrub starts 2024-03-17T23:51:38.755 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:38 smithi191 ceph-mon[36708]: purged_snaps scrub ok 2024-03-17T23:51:38.756 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:38 smithi191 ceph-mon[36708]: osd.6 [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073] boot 2024-03-17T23:51:38.756 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:38 smithi191 ceph-mon[36708]: osdmap e46: 7 total, 7 up, 7 in 2024-03-17T23:51:38.756 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:38 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-17T23:51:38.833 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-17T23:51:38.833 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-17T23:51:38.833 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-03-17T23:51:38.833 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-03-17T23:51:38.833 INFO:teuthology.orchestra.run.smithi079.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126956 s, 826 MB/s 2024-03-17T23:51:38.833 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-03-17T23:51:39.511 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi079:vg_nvme/lv_2 2024-03-17T23:51:39.772 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:39 smithi079 ceph-mon[36241]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:39.772 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:39 smithi079 ceph-mon[36241]: osdmap e47: 7 total, 7 up, 7 in 2024-03-17T23:51:39.789 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.b/config 2024-03-17T23:51:39.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:39 smithi191 ceph-mon[36708]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:39.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:39 smithi191 ceph-mon[36708]: osdmap e47: 7 total, 7 up, 7 in 2024-03-17T23:51:39.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:39 smithi012 ceph-mon[31541]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:39.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:39 smithi012 ceph-mon[31541]: osdmap e47: 7 total, 7 up, 7 in 2024-03-17T23:51:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.430+0000 7f8c9d486700 1 -- 172.21.15.79:0/2881215463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c98108060 msgr2=0x7f8c9810a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:51:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.430+0000 7f8c9d486700 1 --2- 172.21.15.79:0/2881215463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c98108060 0x7f8c9810a4d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000b0a0 tx=0x7f8c9000b3b0 comp rx=0 tx=0).stop 2024-03-17T23:51:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.430+0000 7f8c9d486700 1 -- 172.21.15.79:0/2881215463 shutdown_connections 2024-03-17T23:51:41.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.430+0000 7f8c9d486700 1 --2- 172.21.15.79:0/2881215463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c98108060 0x7f8c9810a4d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.430+0000 7f8c9d486700 1 --2- 172.21.15.79:0/2881215463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c98102e00 0x7f8c981051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.430+0000 7f8c9d486700 1 --2- 172.21.15.79:0/2881215463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c98105730 0x7f8c98107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.430+0000 7f8c9d486700 1 -- 172.21.15.79:0/2881215463 >> 172.21.15.79:0/2881215463 conn(0x7f8c980fc830 msgr2=0x7f8c980fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:51:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.431+0000 7f8c9d486700 1 -- 172.21.15.79:0/2881215463 shutdown_connections 2024-03-17T23:51:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.431+0000 7f8c9d486700 1 -- 172.21.15.79:0/2881215463 wait complete. 2024-03-17T23:51:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.431+0000 7f8c9d486700 1 Processor -- start 2024-03-17T23:51:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.431+0000 7f8c9d486700 1 -- start start 2024-03-17T23:51:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c9d486700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c98102e00 0x7f8c98198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:51:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c9d486700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c98105730 0x7f8c98198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:51:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c9d486700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c98108060 0x7f8c9819fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:51:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c9d486700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9810dfb0 con 0x7f8c98108060 2024-03-17T23:51:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c9d486700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9810de30 con 0x7f8c98105730 2024-03-17T23:51:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c9d486700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9810e130 con 0x7f8c98102e00 2024-03-17T23:51:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c967fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c98105730 0x7f8c98198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:51:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c967fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c98105730 0x7f8c98198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42636/0 (socket says 172.21.15.79:42636) 2024-03-17T23:51:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c967fc700 1 -- 172.21.15.79:0/2027631699 learned_addr learned my addr 172.21.15.79:0/2027631699 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-03-17T23:51:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c96ffd700 1 --2- 172.21.15.79:0/2027631699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c98102e00 0x7f8c98198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:51:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c967fc700 1 -- 172.21.15.79:0/2027631699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c98102e00 msgr2=0x7f8c98198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:51:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c967fc700 1 --2- 172.21.15.79:0/2027631699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c98102e00 0x7f8c98198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c967fc700 1 -- 172.21.15.79:0/2027631699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c98108060 msgr2=0x7f8c9819fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:51:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c967fc700 1 --2- 172.21.15.79:0/2027631699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c98108060 0x7f8c9819fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:51:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c967fc700 1 -- 172.21.15.79:0/2027631699 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c981a01f0 con 0x7f8c98105730 2024-03-17T23:51:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c967fc700 1 --2- 172.21.15.79:0/2027631699 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c98105730 0x7f8c98198ad0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c00b530 tx=0x7f8c8c00b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:51:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c7ffff700 1 -- 172.21.15.79:0/2027631699 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c015020 con 0x7f8c98105730 2024-03-17T23:51:41.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c9d486700 1 -- 172.21.15.79:0/2027631699 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c981a04e0 con 0x7f8c98105730 2024-03-17T23:51:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c7ffff700 1 -- 172.21.15.79:0/2027631699 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c8c011d10 con 0x7f8c98105730 2024-03-17T23:51:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c7ffff700 1 -- 172.21.15.79:0/2027631699 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c00f2e0 con 0x7f8c98105730 2024-03-17T23:51:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.432+0000 7f8c9d486700 1 -- 172.21.15.79:0/2027631699 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c981a0aa0 con 0x7f8c98105730 2024-03-17T23:51:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.433+0000 7f8c9d486700 1 -- 172.21.15.79:0/2027631699 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c98068f90 con 0x7f8c98105730 2024-03-17T23:51:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.435+0000 7f8c7ffff700 1 -- 172.21.15.79:0/2027631699 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8c8c018020 con 0x7f8c98105730 2024-03-17T23:51:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.436+0000 7f8c7ffff700 1 --2- 172.21.15.79:0/2027631699 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8c800a2890 0x7f8c800a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:51:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.436+0000 7f8c7ffff700 1 -- 172.21.15.79:0/2027631699 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f8c8c0cdb30 con 0x7f8c98105730 2024-03-17T23:51:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.436+0000 7f8c96ffd700 1 --2- 172.21.15.79:0/2027631699 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8c800a2890 0x7f8c800a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:51:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.437+0000 7f8c96ffd700 1 --2- 172.21.15.79:0/2027631699 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8c800a2890 0x7f8c800a4d50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c88006fd0 tx=0x7f8c88008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:51:41.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.439+0000 7f8c7ffff700 1 -- 172.21.15.79:0/2027631699 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8c8c01f020 con 0x7f8c98105730 2024-03-17T23:51:41.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:51:41.650+0000 7f8c9d486700 1 -- 172.21.15.79:0/2027631699 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8c98062710 con 0x7f8c800a2890 2024-03-17T23:51:41.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:41 smithi079 ceph-mon[36241]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:41.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:41 smithi191 ceph-mon[36708]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:41.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:41 smithi012 ceph-mon[31541]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:42.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: Detected new or changed devices on smithi012 2024-03-17T23:51:42.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:42.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:42.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:42.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:51:42.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:42.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:51:42.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:42.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi012 to 5698M 2024-03-17T23:51:42.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:51:42.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:42.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:42.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:42.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:51:42.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:42 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:43.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: Detected new or changed devices on smithi012 2024-03-17T23:51:43.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:43.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:43.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:43.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:51:43.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:43.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:51:43.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:43.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi012 to 5698M 2024-03-17T23:51:43.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:51:43.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:43.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:43.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:43.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:51:43.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:42 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:43.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: Detected new or changed devices on smithi012 2024-03-17T23:51:43.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:43.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:43.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:43.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:51:43.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:43.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:51:43.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:51:43.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi012 to 5698M 2024-03-17T23:51:43.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:51:43.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:43.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:43.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:43.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:51:43.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:42 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:44.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:43 smithi079 ceph-mon[36241]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:44.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:43 smithi191 ceph-mon[36708]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:44.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:43 smithi012 ceph-mon[31541]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:45.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:44 smithi079 ceph-mon[36241]: from='client.? 172.21.15.79:0/2856315651' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f77cf43-cd10-4ef1-bb88-dbe4f154c353"}]: dispatch 2024-03-17T23:51:45.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:44 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f77cf43-cd10-4ef1-bb88-dbe4f154c353"}]: dispatch 2024-03-17T23:51:45.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:44 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f77cf43-cd10-4ef1-bb88-dbe4f154c353"}]': finished 2024-03-17T23:51:45.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:44 smithi079 ceph-mon[36241]: osdmap e48: 8 total, 7 up, 8 in 2024-03-17T23:51:45.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-17T23:51:45.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:44 smithi191 ceph-mon[36708]: from='client.? 172.21.15.79:0/2856315651' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f77cf43-cd10-4ef1-bb88-dbe4f154c353"}]: dispatch 2024-03-17T23:51:45.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:44 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f77cf43-cd10-4ef1-bb88-dbe4f154c353"}]: dispatch 2024-03-17T23:51:45.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:44 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f77cf43-cd10-4ef1-bb88-dbe4f154c353"}]': finished 2024-03-17T23:51:45.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:44 smithi191 ceph-mon[36708]: osdmap e48: 8 total, 7 up, 8 in 2024-03-17T23:51:45.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-17T23:51:45.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:44 smithi012 ceph-mon[31541]: from='client.? 172.21.15.79:0/2856315651' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f77cf43-cd10-4ef1-bb88-dbe4f154c353"}]: dispatch 2024-03-17T23:51:45.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:44 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f77cf43-cd10-4ef1-bb88-dbe4f154c353"}]: dispatch 2024-03-17T23:51:45.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:44 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f77cf43-cd10-4ef1-bb88-dbe4f154c353"}]': finished 2024-03-17T23:51:45.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:44 smithi012 ceph-mon[31541]: osdmap e48: 8 total, 7 up, 8 in 2024-03-17T23:51:45.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-17T23:51:45.968 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:45 smithi079 ceph-mon[36241]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:45.968 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:45 smithi079 ceph-mon[36241]: from='client.? 172.21.15.79:0/2935396294' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:51:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:45 smithi191 ceph-mon[36708]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:45 smithi191 ceph-mon[36708]: from='client.? 172.21.15.79:0/2935396294' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:51:46.136 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:45 smithi012 ceph-mon[31541]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:46.136 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:45 smithi012 ceph-mon[31541]: from='client.? 172.21.15.79:0/2935396294' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:51:48.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:47 smithi079 ceph-mon[36241]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:48.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:47 smithi191 ceph-mon[36708]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:48.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:47 smithi012 ceph-mon[31541]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:49.993 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:49 smithi191 ceph-mon[36708]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:50.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:49 smithi079 ceph-mon[36241]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:50.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:49 smithi012 ceph-mon[31541]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:52.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:51 smithi079 ceph-mon[36241]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:52.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-17T23:51:52.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:52.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:51 smithi191 ceph-mon[36708]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:52.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-17T23:51:52.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:52.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:51 smithi012 ceph-mon[31541]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:52.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-17T23:51:52.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:51:53.014 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:52 smithi079 ceph-mon[36241]: Deploying daemon osd.7 on smithi079 2024-03-17T23:51:53.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:52 smithi191 ceph-mon[36708]: Deploying daemon osd.7 on smithi079 2024-03-17T23:51:53.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:52 smithi012 ceph-mon[31541]: Deploying daemon osd.7 on smithi079 2024-03-17T23:51:53.975 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:53 smithi079 ceph-mon[36241]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:54.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:53 smithi191 ceph-mon[36708]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:54.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:53 smithi012 ceph-mon[31541]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:55.992 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:55 smithi079 ceph-mon[36241]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:56.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:55 smithi191 ceph-mon[36708]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:56.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:55 smithi012 ceph-mon[31541]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:58.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:57 smithi079 ceph-mon[36241]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:58.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:58.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:58.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:51:58.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:57 smithi191 ceph-mon[36708]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:58.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:58.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:58.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:51:58.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:57 smithi012 ceph-mon[31541]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:51:58.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:58.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:51:58.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:52:00.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:51:59 smithi079 ceph-mon[36241]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:52:00.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:51:59 smithi191 ceph-mon[36708]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:52:00.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:51:59 smithi012 ceph-mon[31541]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:52:01.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:52:01.035+0000 7f8c7ffff700 1 -- 172.21.15.79:0/2027631699 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8c98062710 con 0x7f8c800a2890 2024-03-17T23:52:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 7 on host 'smithi079' 2024-03-17T23:52:01.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:52:01.038+0000 7f8c9d486700 1 -- 172.21.15.79:0/2027631699 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8c800a2890 msgr2=0x7f8c800a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:52:01.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:52:01.038+0000 7f8c9d486700 1 --2- 172.21.15.79:0/2027631699 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8c800a2890 0x7f8c800a4d50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c88006fd0 tx=0x7f8c88008040 comp rx=0 tx=0).stop 2024-03-17T23:52:01.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:52:01.038+0000 7f8c9d486700 1 -- 172.21.15.79:0/2027631699 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c98105730 msgr2=0x7f8c98198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:52:01.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:52:01.038+0000 7f8c9d486700 1 --2- 172.21.15.79:0/2027631699 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c98105730 0x7f8c98198ad0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c00b530 tx=0x7f8c8c00b8f0 comp rx=0 tx=0).stop 2024-03-17T23:52:01.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:52:01.038+0000 7f8c9d486700 1 -- 172.21.15.79:0/2027631699 shutdown_connections 2024-03-17T23:52:01.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:52:01.038+0000 7f8c9d486700 1 --2- 172.21.15.79:0/2027631699 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8c800a2890 0x7f8c800a4d50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:01.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:52:01.038+0000 7f8c9d486700 1 --2- 172.21.15.79:0/2027631699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c98108060 0x7f8c9819fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:01.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:52:01.038+0000 7f8c9d486700 1 --2- 172.21.15.79:0/2027631699 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c98102e00 0x7f8c98198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:01.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:52:01.038+0000 7f8c9d486700 1 --2- 172.21.15.79:0/2027631699 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c98105730 0x7f8c98198ad0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:01.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:52:01.038+0000 7f8c9d486700 1 -- 172.21.15.79:0/2027631699 >> 172.21.15.79:0/2027631699 conn(0x7f8c980fc830 msgr2=0x7f8c981036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:52:01.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:52:01.038+0000 7f8c9d486700 1 -- 172.21.15.79:0/2027631699 shutdown_connections 2024-03-17T23:52:01.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:52:01.038+0000 7f8c9d486700 1 -- 172.21.15.79:0/2027631699 wait complete. 2024-03-17T23:52:01.801 DEBUG:teuthology.orchestra.run.smithi079:osd.7> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.7.service 2024-03-17T23:52:01.803 INFO:tasks.cephadm:Deploying osd.8 on smithi191 with /dev/vg_nvme/lv_2... 2024-03-17T23:52:01.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-17T23:52:02.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:01 smithi079 ceph-mon[36241]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:52:02.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:01 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:02.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:01 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:02.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:01 smithi079 ceph-mon[36241]: from='osd.7 [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-17T23:52:02.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:01 smithi079 ceph-mon[36241]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-17T23:52:02.054 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-17T23:52:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:01 smithi191 ceph-mon[36708]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:52:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:01 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:01 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:01 smithi191 ceph-mon[36708]: from='osd.7 [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-17T23:52:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:01 smithi191 ceph-mon[36708]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-17T23:52:02.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:01 smithi012 ceph-mon[31541]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:52:02.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:01 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:02.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:01 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:02.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:01 smithi012 ceph-mon[31541]: from='osd.7 [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-17T23:52:02.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:01 smithi012 ceph-mon[31541]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-17T23:52:03.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:02 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:03.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:02 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:03.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:02 smithi012 ceph-mon[31541]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-17T23:52:03.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:02 smithi012 ceph-mon[31541]: from='osd.7 [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:52:03.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:02 smithi012 ceph-mon[31541]: osdmap e49: 8 total, 7 up, 8 in 2024-03-17T23:52:03.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:02 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-17T23:52:03.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:02 smithi012 ceph-mon[31541]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:52:03.177 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:02 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:03.177 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:02 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:03.177 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:02 smithi191 ceph-mon[36708]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-17T23:52:03.177 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:02 smithi191 ceph-mon[36708]: from='osd.7 [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:52:03.177 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:02 smithi191 ceph-mon[36708]: osdmap e49: 8 total, 7 up, 8 in 2024-03-17T23:52:03.177 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:02 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-17T23:52:03.177 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:02 smithi191 ceph-mon[36708]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:52:03.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:02 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:03.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:02 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:03.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:02 smithi079 ceph-mon[36241]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-17T23:52:03.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:02 smithi079 ceph-mon[36241]: from='osd.7 [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:52:03.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:02 smithi079 ceph-mon[36241]: osdmap e49: 8 total, 7 up, 8 in 2024-03-17T23:52:03.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:02 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-17T23:52:03.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:02 smithi079 ceph-mon[36241]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:52:03.296 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 17 23:52:03 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-17T23:52:03.015+0000 7f6fefdb1700 -1 osd.7 0 waiting for initial osdmap 2024-03-17T23:52:03.296 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 17 23:52:03 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-17T23:52:03.024+0000 7f6febba4700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:52:04.069 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-17T23:52:04.069 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-17T23:52:04.070 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-03-17T23:52:04.070 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-03-17T23:52:04.070 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113716 s, 922 MB/s 2024-03-17T23:52:04.070 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-03-17T23:52:04.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:03 smithi191 ceph-mon[36708]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:52:04.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:03 smithi191 ceph-mon[36708]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-17T23:52:04.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:03 smithi191 ceph-mon[36708]: osdmap e50: 8 total, 7 up, 8 in 2024-03-17T23:52:04.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-17T23:52:04.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:03 smithi191 ceph-mon[36708]: osd.7 [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856] boot 2024-03-17T23:52:04.095 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:03 smithi191 ceph-mon[36708]: osdmap e51: 8 total, 8 up, 8 in 2024-03-17T23:52:04.095 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-17T23:52:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:03 smithi012 ceph-mon[31541]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:52:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:03 smithi012 ceph-mon[31541]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-17T23:52:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:03 smithi012 ceph-mon[31541]: osdmap e50: 8 total, 7 up, 8 in 2024-03-17T23:52:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-17T23:52:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:03 smithi012 ceph-mon[31541]: osd.7 [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856] boot 2024-03-17T23:52:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:03 smithi012 ceph-mon[31541]: osdmap e51: 8 total, 8 up, 8 in 2024-03-17T23:52:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-17T23:52:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:03 smithi079 ceph-mon[36241]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-17T23:52:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:03 smithi079 ceph-mon[36241]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-17T23:52:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:03 smithi079 ceph-mon[36241]: osdmap e50: 8 total, 7 up, 8 in 2024-03-17T23:52:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:03 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-17T23:52:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:03 smithi079 ceph-mon[36241]: osd.7 [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856] boot 2024-03-17T23:52:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:03 smithi079 ceph-mon[36241]: osdmap e51: 8 total, 8 up, 8 in 2024-03-17T23:52:04.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:03 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-17T23:52:04.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi191:vg_nvme/lv_2 2024-03-17T23:52:05.002 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-17T23:52:05.406 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:05 smithi191 ceph-mon[36708]: purged_snaps scrub starts 2024-03-17T23:52:05.406 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:05 smithi191 ceph-mon[36708]: purged_snaps scrub ok 2024-03-17T23:52:05.406 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:05 smithi191 ceph-mon[36708]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:05.406 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:05 smithi191 ceph-mon[36708]: osdmap e52: 8 total, 8 up, 8 in 2024-03-17T23:52:05.406 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:05.406 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:05.406 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:05.406 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:05.407 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:05.407 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:05.407 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:05.407 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:52:05.407 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:05 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:05.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:05 smithi079 ceph-mon[36241]: purged_snaps scrub starts 2024-03-17T23:52:05.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:05 smithi079 ceph-mon[36241]: purged_snaps scrub ok 2024-03-17T23:52:05.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:05 smithi079 ceph-mon[36241]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:05.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:05 smithi079 ceph-mon[36241]: osdmap e52: 8 total, 8 up, 8 in 2024-03-17T23:52:05.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:05.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:52:05.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:05 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:05.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:05 smithi012 ceph-mon[31541]: purged_snaps scrub starts 2024-03-17T23:52:05.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:05 smithi012 ceph-mon[31541]: purged_snaps scrub ok 2024-03-17T23:52:05.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:05 smithi012 ceph-mon[31541]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:05.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:05 smithi012 ceph-mon[31541]: osdmap e52: 8 total, 8 up, 8 in 2024-03-17T23:52:05.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:05.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:05.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:05.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:05.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:05.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:05.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:05.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:52:05.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:05 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:06.490 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:06 smithi191 ceph-mon[36708]: Detected new or changed devices on smithi079 2024-03-17T23:52:06.490 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:06 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi079 to 5693M 2024-03-17T23:52:06.490 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:06 smithi191 ceph-mon[36708]: osdmap e53: 8 total, 8 up, 8 in 2024-03-17T23:52:06.490 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:52:06.490 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:06.490 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:52:06.490 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:06.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:06 smithi012 ceph-mon[31541]: Detected new or changed devices on smithi079 2024-03-17T23:52:06.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:06 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi079 to 5693M 2024-03-17T23:52:06.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:06 smithi012 ceph-mon[31541]: osdmap e53: 8 total, 8 up, 8 in 2024-03-17T23:52:06.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:52:06.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:06.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:52:06.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:06.725 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.723+0000 7fce21025700 1 -- 172.21.15.191:0/1891029745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce1c103e90 msgr2=0x7fce1c10a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:52:06.725 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.723+0000 7fce21025700 1 --2- 172.21.15.191:0/1891029745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce1c103e90 0x7fce1c10a740 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fce1400b0a0 tx=0x7fce1400b3b0 comp rx=0 tx=0).stop 2024-03-17T23:52:06.725 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.723+0000 7fce21025700 1 -- 172.21.15.191:0/1891029745 shutdown_connections 2024-03-17T23:52:06.725 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.723+0000 7fce21025700 1 --2- 172.21.15.191:0/1891029745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce1c103e90 0x7fce1c10a740 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:06.725 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.723+0000 7fce21025700 1 --2- 172.21.15.191:0/1891029745 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c1022f0 0x7fce1c102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:06.725 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.723+0000 7fce21025700 1 --2- 172.21.15.191:0/1891029745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce1c1034f0 0x7fce1c103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:06.725 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.723+0000 7fce21025700 1 -- 172.21.15.191:0/1891029745 >> 172.21.15.191:0/1891029745 conn(0x7fce1c0fd8b0 msgr2=0x7fce1c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:52:06.725 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.723+0000 7fce21025700 1 -- 172.21.15.191:0/1891029745 shutdown_connections 2024-03-17T23:52:06.725 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.723+0000 7fce21025700 1 -- 172.21.15.191:0/1891029745 wait complete. 2024-03-17T23:52:06.725 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.724+0000 7fce21025700 1 Processor -- start 2024-03-17T23:52:06.727 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.724+0000 7fce21025700 1 -- start start 2024-03-17T23:52:06.727 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.724+0000 7fce21025700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce1c1022f0 0x7fce1c198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:52:06.727 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.724+0000 7fce21025700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce1c1034f0 0x7fce1c198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:52:06.727 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.724+0000 7fce21025700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c103e90 0x7fce1c19f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.724+0000 7fce21025700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fce1c10df30 con 0x7fce1c1034f0 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.724+0000 7fce21025700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fce1c10ddb0 con 0x7fce1c103e90 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.724+0000 7fce21025700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fce1c10e0b0 con 0x7fce1c1022f0 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce1ad9d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce1c1022f0 0x7fce1c198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce1ad9d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce1c1022f0 0x7fce1c198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:34578/0 (socket says 172.21.15.191:34578) 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce1ad9d700 1 -- 172.21.15.191:0/1040027537 learned_addr learned my addr 172.21.15.191:0/1040027537 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce1b59e700 1 --2- 172.21.15.191:0/1040027537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c103e90 0x7fce1c19f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce1ad9d700 1 -- 172.21.15.191:0/1040027537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c103e90 msgr2=0x7fce1c19f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce1ad9d700 1 --2- 172.21.15.191:0/1040027537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c103e90 0x7fce1c19f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce1ad9d700 1 -- 172.21.15.191:0/1040027537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce1c1034f0 msgr2=0x7fce1c198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce1ad9d700 1 --2- 172.21.15.191:0/1040027537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce1c1034f0 0x7fce1c198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce1ad9d700 1 -- 172.21.15.191:0/1040027537 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1c1a0080 con 0x7fce1c1022f0 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce1ad9d700 1 --2- 172.21.15.191:0/1040027537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce1c1022f0 0x7fce1c198420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00c9c0 tx=0x7fce0c00ccd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce03fff700 1 -- 172.21.15.191:0/1040027537 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c007ac0 con 0x7fce1c1022f0 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce21025700 1 -- 172.21.15.191:0/1040027537 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce1c1a0370 con 0x7fce1c1022f0 2024-03-17T23:52:06.728 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce03fff700 1 -- 172.21.15.191:0/1040027537 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce0c007c60 con 0x7fce1c1022f0 2024-03-17T23:52:06.729 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce03fff700 1 -- 172.21.15.191:0/1040027537 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c018660 con 0x7fce1c1022f0 2024-03-17T23:52:06.729 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.725+0000 7fce21025700 1 -- 172.21.15.191:0/1040027537 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce1c1a0930 con 0x7fce1c1022f0 2024-03-17T23:52:06.729 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.726+0000 7fce21025700 1 -- 172.21.15.191:0/1040027537 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce1c068f90 con 0x7fce1c1022f0 2024-03-17T23:52:06.729 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.727+0000 7fce03fff700 1 -- 172.21.15.191:0/1040027537 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fce0c004020 con 0x7fce1c1022f0 2024-03-17T23:52:06.729 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.728+0000 7fce03fff700 1 --2- 172.21.15.191:0/1040027537 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fce040a2890 0x7fce040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:52:06.729 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.728+0000 7fce03fff700 1 -- 172.21.15.191:0/1040027537 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5397+0+0 (secure 0 0 0) 0x7fce0c0ce050 con 0x7fce1c1022f0 2024-03-17T23:52:06.730 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.728+0000 7fce1a59c700 1 --2- 172.21.15.191:0/1040027537 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fce040a2890 0x7fce040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:52:06.730 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.729+0000 7fce1a59c700 1 --2- 172.21.15.191:0/1040027537 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fce040a2890 0x7fce040a4d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fce100097b0 tx=0x7fce10006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:52:06.733 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.732+0000 7fce03fff700 1 -- 172.21.15.191:0/1040027537 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fce0c014030 con 0x7fce1c1022f0 2024-03-17T23:52:06.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:06 smithi079 ceph-mon[36241]: Detected new or changed devices on smithi079 2024-03-17T23:52:06.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:06 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi079 to 5693M 2024-03-17T23:52:06.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:06 smithi079 ceph-mon[36241]: osdmap e53: 8 total, 8 up, 8 in 2024-03-17T23:52:06.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:52:06.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:06.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:52:06.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:06.936 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:06.935+0000 7fce21025700 1 -- 172.21.15.191:0/1040027537 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fce1c062710 con 0x7fce040a2890 2024-03-17T23:52:07.606 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:07 smithi191 ceph-mon[36708]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:07.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:07 smithi191 ceph-mon[36708]: osdmap e54: 8 total, 8 up, 8 in 2024-03-17T23:52:07.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:07 smithi191 ceph-mon[36708]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:52:07.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:07 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:52:07.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:07 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:52:07.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:07 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:07.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:07 smithi012 ceph-mon[31541]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:07.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:07 smithi012 ceph-mon[31541]: osdmap e54: 8 total, 8 up, 8 in 2024-03-17T23:52:07.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:07 smithi012 ceph-mon[31541]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:52:07.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:07 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:52:07.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:07 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:52:07.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:07 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:07.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:07 smithi079 ceph-mon[36241]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:07.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:07 smithi079 ceph-mon[36241]: osdmap e54: 8 total, 8 up, 8 in 2024-03-17T23:52:07.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:07 smithi079 ceph-mon[36241]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:52:07.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:07 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:52:07.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:07 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:52:07.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:07 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:08.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:08 smithi012 ceph-mon[31541]: osdmap e55: 8 total, 8 up, 8 in 2024-03-17T23:52:08.756 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:08 smithi191 ceph-mon[36708]: osdmap e55: 8 total, 8 up, 8 in 2024-03-17T23:52:08.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:08 smithi079 ceph-mon[36241]: osdmap e55: 8 total, 8 up, 8 in 2024-03-17T23:52:09.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:09 smithi012 ceph-mon[31541]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:09.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:09 smithi079 ceph-mon[36241]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:09 smithi191 ceph-mon[36708]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:10.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:10 smithi012 ceph-mon[31541]: from='client.? 172.21.15.191:0/764283110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ae081be-d65c-456a-81f5-8a299254b8ae"}]: dispatch 2024-03-17T23:52:10.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:10 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ae081be-d65c-456a-81f5-8a299254b8ae"}]: dispatch 2024-03-17T23:52:10.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:10 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ae081be-d65c-456a-81f5-8a299254b8ae"}]': finished 2024-03-17T23:52:10.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:10 smithi012 ceph-mon[31541]: osdmap e56: 9 total, 8 up, 9 in 2024-03-17T23:52:10.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:10 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-17T23:52:10.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:10 smithi079 ceph-mon[36241]: from='client.? 172.21.15.191:0/764283110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ae081be-d65c-456a-81f5-8a299254b8ae"}]: dispatch 2024-03-17T23:52:10.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:10 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ae081be-d65c-456a-81f5-8a299254b8ae"}]: dispatch 2024-03-17T23:52:10.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:10 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ae081be-d65c-456a-81f5-8a299254b8ae"}]': finished 2024-03-17T23:52:10.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:10 smithi079 ceph-mon[36241]: osdmap e56: 9 total, 8 up, 9 in 2024-03-17T23:52:10.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:10 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-17T23:52:10.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:10 smithi191 ceph-mon[36708]: from='client.? 172.21.15.191:0/764283110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ae081be-d65c-456a-81f5-8a299254b8ae"}]: dispatch 2024-03-17T23:52:10.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:10 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ae081be-d65c-456a-81f5-8a299254b8ae"}]: dispatch 2024-03-17T23:52:10.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:10 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ae081be-d65c-456a-81f5-8a299254b8ae"}]': finished 2024-03-17T23:52:10.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:10 smithi191 ceph-mon[36708]: osdmap e56: 9 total, 8 up, 9 in 2024-03-17T23:52:10.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:10 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-17T23:52:11.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:11 smithi012 ceph-mon[31541]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:11.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:11 smithi012 ceph-mon[31541]: from='client.? 172.21.15.191:0/3036692867' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:52:11.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:11 smithi079 ceph-mon[36241]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:11.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:11 smithi079 ceph-mon[36241]: from='client.? 172.21.15.191:0/3036692867' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:52:11.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:11 smithi191 ceph-mon[36708]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:11.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:11 smithi191 ceph-mon[36708]: from='client.? 172.21.15.191:0/3036692867' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:52:13.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:13 smithi012 ceph-mon[31541]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:13.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:13 smithi191 ceph-mon[36708]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:13.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:13 smithi079 ceph-mon[36241]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:15.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:15 smithi012 ceph-mon[31541]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:15.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:15 smithi191 ceph-mon[36708]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:15.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:15 smithi079 ceph-mon[36241]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:17.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:17 smithi191 ceph-mon[36708]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:17.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-17T23:52:17.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:17.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:17 smithi012 ceph-mon[31541]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:17.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-17T23:52:17.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:17.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:17 smithi079 ceph-mon[36241]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:17.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-17T23:52:17.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:18.588 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:18 smithi191 ceph-mon[36708]: Deploying daemon osd.8 on smithi191 2024-03-17T23:52:18.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:18 smithi012 ceph-mon[31541]: Deploying daemon osd.8 on smithi191 2024-03-17T23:52:18.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:18 smithi079 ceph-mon[36241]: Deploying daemon osd.8 on smithi191 2024-03-17T23:52:19.576 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:19 smithi191 ceph-mon[36708]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:19.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:19 smithi012 ceph-mon[31541]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:19.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:19 smithi079 ceph-mon[36241]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:21.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:21 smithi191 ceph-mon[36708]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:21.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:21 smithi012 ceph-mon[31541]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:21.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:21 smithi079 ceph-mon[36241]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:23.730 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:23 smithi191 ceph-mon[36708]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:23.730 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:23 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:23.730 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:23 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:23.730 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:23 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:52:23.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:23 smithi079 ceph-mon[36241]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:23.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:23 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:23.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:23 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:23.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:23 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:52:23.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:23 smithi012 ceph-mon[31541]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:23.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:23 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:23.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:23 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:23.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:23 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:52:25.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:25 smithi079 ceph-mon[36241]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:25.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:25 smithi191 ceph-mon[36708]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:25.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:25 smithi012 ceph-mon[31541]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:27.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:27.025+0000 7fce03fff700 1 -- 172.21.15.191:0/1040027537 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fce1c062710 con 0x7fce040a2890 2024-03-17T23:52:27.026 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 8 on host 'smithi191' 2024-03-17T23:52:27.028 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:27.028+0000 7fce21025700 1 -- 172.21.15.191:0/1040027537 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fce040a2890 msgr2=0x7fce040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:52:27.028 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:27.028+0000 7fce21025700 1 --2- 172.21.15.191:0/1040027537 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fce040a2890 0x7fce040a4d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fce100097b0 tx=0x7fce10006d20 comp rx=0 tx=0).stop 2024-03-17T23:52:27.028 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:27.028+0000 7fce21025700 1 -- 172.21.15.191:0/1040027537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce1c1022f0 msgr2=0x7fce1c198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:52:27.028 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:27.028+0000 7fce21025700 1 --2- 172.21.15.191:0/1040027537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce1c1022f0 0x7fce1c198420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00c9c0 tx=0x7fce0c00ccd0 comp rx=0 tx=0).stop 2024-03-17T23:52:27.028 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:27.028+0000 7fce21025700 1 -- 172.21.15.191:0/1040027537 shutdown_connections 2024-03-17T23:52:27.028 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:27.028+0000 7fce21025700 1 --2- 172.21.15.191:0/1040027537 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fce040a2890 0x7fce040a4d50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:27.028 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:27.028+0000 7fce21025700 1 --2- 172.21.15.191:0/1040027537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c103e90 0x7fce1c19f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:27.028 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:27.028+0000 7fce21025700 1 --2- 172.21.15.191:0/1040027537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce1c1022f0 0x7fce1c198420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:27.028 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:27.028+0000 7fce21025700 1 --2- 172.21.15.191:0/1040027537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce1c1034f0 0x7fce1c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:27.029 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:27.028+0000 7fce21025700 1 -- 172.21.15.191:0/1040027537 >> 172.21.15.191:0/1040027537 conn(0x7fce1c0fd8b0 msgr2=0x7fce1c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:52:27.029 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:27.028+0000 7fce21025700 1 -- 172.21.15.191:0/1040027537 shutdown_connections 2024-03-17T23:52:27.029 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:52:27.028+0000 7fce21025700 1 -- 172.21.15.191:0/1040027537 wait complete. 2024-03-17T23:52:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:27 smithi191 ceph-mon[36708]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:27 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:27 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:27 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:27.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:27 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:52:27.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:27 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:27 smithi191 ceph-mon[36708]: from='osd.8 [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-17T23:52:27.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:27 smithi191 ceph-mon[36708]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-17T23:52:27.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:27 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:27 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:52:27.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:27 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:27 smithi012 ceph-mon[31541]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:27.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:27.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:52:27.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:27 smithi012 ceph-mon[31541]: from='osd.8 [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-17T23:52:27.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:27 smithi012 ceph-mon[31541]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-17T23:52:27.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:52:27.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:27 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:27 smithi079 ceph-mon[36241]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:27.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:27 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:27 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:27 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:27.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:27 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:52:27.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:27 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:27 smithi079 ceph-mon[36241]: from='osd.8 [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-17T23:52:27.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:27 smithi079 ceph-mon[36241]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-17T23:52:27.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:27 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:27 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:52:27.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:27 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:27.751 DEBUG:teuthology.orchestra.run.smithi191:osd.8> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.8.service 2024-03-17T23:52:27.754 INFO:tasks.cephadm:Deploying osd.9 on smithi012 with /dev/vg_nvme/lv_1... 2024-03-17T23:52:27.754 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-17T23:52:28.232 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:52:28.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:28 smithi012 ceph-mon[31541]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-17T23:52:28.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:28 smithi012 ceph-mon[31541]: from='osd.8 [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:52:28.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:28 smithi012 ceph-mon[31541]: osdmap e57: 9 total, 8 up, 9 in 2024-03-17T23:52:28.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-17T23:52:28.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:28 smithi012 ceph-mon[31541]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:52:28.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:28 smithi079 ceph-mon[36241]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-17T23:52:28.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:28 smithi079 ceph-mon[36241]: from='osd.8 [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:52:28.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:28 smithi079 ceph-mon[36241]: osdmap e57: 9 total, 8 up, 9 in 2024-03-17T23:52:28.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-17T23:52:28.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:28 smithi079 ceph-mon[36241]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:52:28.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:28 smithi191 ceph-mon[36708]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-17T23:52:28.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:28 smithi191 ceph-mon[36708]: from='osd.8 [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:52:28.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:28 smithi191 ceph-mon[36708]: osdmap e57: 9 total, 8 up, 9 in 2024-03-17T23:52:28.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-17T23:52:28.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:28 smithi191 ceph-mon[36708]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:52:28.842 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 17 23:52:28 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-17T23:52:28.444+0000 7f187d44c700 -1 osd.8 0 waiting for initial osdmap 2024-03-17T23:52:28.842 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 17 23:52:28 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-17T23:52:28.453+0000 7f187923f700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:52:29.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:29 smithi191 ceph-mon[36708]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:29.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:29 smithi191 ceph-mon[36708]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-17T23:52:29.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:29 smithi191 ceph-mon[36708]: osdmap e58: 9 total, 8 up, 9 in 2024-03-17T23:52:29.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:29 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-17T23:52:29.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:29 smithi012 ceph-mon[31541]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:29.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:29 smithi012 ceph-mon[31541]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-17T23:52:29.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:29 smithi012 ceph-mon[31541]: osdmap e58: 9 total, 8 up, 9 in 2024-03-17T23:52:29.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:29 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-17T23:52:29.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:29 smithi079 ceph-mon[36241]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-17T23:52:29.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:29 smithi079 ceph-mon[36241]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-17T23:52:29.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:29 smithi079 ceph-mon[36241]: osdmap e58: 9 total, 8 up, 9 in 2024-03-17T23:52:29.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:29 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-17T23:52:30.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:30 smithi191 ceph-mon[36708]: purged_snaps scrub starts 2024-03-17T23:52:30.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:30 smithi191 ceph-mon[36708]: purged_snaps scrub ok 2024-03-17T23:52:30.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:30 smithi191 ceph-mon[36708]: osd.8 [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703] boot 2024-03-17T23:52:30.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:30 smithi191 ceph-mon[36708]: osdmap e59: 9 total, 9 up, 9 in 2024-03-17T23:52:30.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:30 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-17T23:52:30.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:30 smithi191 ceph-mon[36708]: osdmap e60: 9 total, 9 up, 9 in 2024-03-17T23:52:30.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:30 smithi012 ceph-mon[31541]: purged_snaps scrub starts 2024-03-17T23:52:30.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:30 smithi012 ceph-mon[31541]: purged_snaps scrub ok 2024-03-17T23:52:30.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:30 smithi012 ceph-mon[31541]: osd.8 [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703] boot 2024-03-17T23:52:30.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:30 smithi012 ceph-mon[31541]: osdmap e59: 9 total, 9 up, 9 in 2024-03-17T23:52:30.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:30 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-17T23:52:30.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:30 smithi012 ceph-mon[31541]: osdmap e60: 9 total, 9 up, 9 in 2024-03-17T23:52:30.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:30 smithi079 ceph-mon[36241]: purged_snaps scrub starts 2024-03-17T23:52:30.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:30 smithi079 ceph-mon[36241]: purged_snaps scrub ok 2024-03-17T23:52:30.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:30 smithi079 ceph-mon[36241]: osd.8 [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703] boot 2024-03-17T23:52:30.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:30 smithi079 ceph-mon[36241]: osdmap e59: 9 total, 9 up, 9 in 2024-03-17T23:52:30.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:30 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-17T23:52:30.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:30 smithi079 ceph-mon[36241]: osdmap e60: 9 total, 9 up, 9 in 2024-03-17T23:52:31.434 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-17T23:52:31.435 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-17T23:52:31.435 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-03-17T23:52:31.435 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-03-17T23:52:31.435 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0354152 s, 296 MB/s 2024-03-17T23:52:31.435 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-03-17T23:52:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:31 smithi079 ceph-mon[36241]: pgmap v217: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:52:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:31 smithi079 ceph-mon[36241]: osdmap e61: 9 total, 9 up, 9 in 2024-03-17T23:52:31.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:31 smithi191 ceph-mon[36708]: pgmap v217: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:31.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:52:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:31 smithi191 ceph-mon[36708]: osdmap e61: 9 total, 9 up, 9 in 2024-03-17T23:52:31.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:31 smithi012 ceph-mon[31541]: pgmap v217: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:31.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:31.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:31.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:31.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:31.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:52:31.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:31.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:31.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:52:31.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:31.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:31 smithi012 ceph-mon[31541]: osdmap e61: 9 total, 9 up, 9 in 2024-03-17T23:52:32.303 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi012:vg_nvme/lv_1 2024-03-17T23:52:32.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:32 smithi012 ceph-mon[31541]: Detected new or changed devices on smithi191 2024-03-17T23:52:32.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:32 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi191 to 5693M 2024-03-17T23:52:32.779 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:52:32.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:32 smithi079 ceph-mon[36241]: Detected new or changed devices on smithi191 2024-03-17T23:52:32.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:32 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi191 to 5693M 2024-03-17T23:52:32.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:32 smithi191 ceph-mon[36708]: Detected new or changed devices on smithi191 2024-03-17T23:52:32.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:32 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi191 to 5693M 2024-03-17T23:52:33.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:33 smithi012 ceph-mon[31541]: pgmap v220: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:33.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:33 smithi012 ceph-mon[31541]: osdmap e62: 9 total, 9 up, 9 in 2024-03-17T23:52:33.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:33 smithi079 ceph-mon[36241]: pgmap v220: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:33.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:33 smithi079 ceph-mon[36241]: osdmap e62: 9 total, 9 up, 9 in 2024-03-17T23:52:33.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:33 smithi191 ceph-mon[36708]: pgmap v220: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:33.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:33 smithi191 ceph-mon[36708]: osdmap e62: 9 total, 9 up, 9 in 2024-03-17T23:52:34.619 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:34 smithi012 ceph-mon[31541]: osdmap e63: 9 total, 9 up, 9 in 2024-03-17T23:52:34.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:34 smithi079 ceph-mon[36241]: osdmap e63: 9 total, 9 up, 9 in 2024-03-17T23:52:34.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:34 smithi191 ceph-mon[36708]: osdmap e63: 9 total, 9 up, 9 in 2024-03-17T23:52:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.125+0000 7f74bd000700 1 -- 172.21.15.12:0/2442750462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74b8076280 msgr2=0x7f74b80746e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:52:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.125+0000 7f74bd000700 1 --2- 172.21.15.12:0/2442750462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74b8076280 0x7f74b80746e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f74a800b0a0 tx=0x7f74a800b3b0 comp rx=0 tx=0).stop 2024-03-17T23:52:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.125+0000 7f74bd000700 1 -- 172.21.15.12:0/2442750462 shutdown_connections 2024-03-17T23:52:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.125+0000 7f74bd000700 1 --2- 172.21.15.12:0/2442750462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74b810a230 0x7f74b810c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.125+0000 7f74bd000700 1 --2- 172.21.15.12:0/2442750462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74b8076280 0x7f74b80746e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.125+0000 7f74bd000700 1 --2- 172.21.15.12:0/2442750462 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74b8074cb0 0x7f74b8075130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.125+0000 7f74bd000700 1 -- 172.21.15.12:0/2442750462 >> 172.21.15.12:0/2442750462 conn(0x7f74b80fd8d0 msgr2=0x7f74b80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:52:35.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.125+0000 7f74bd000700 1 -- 172.21.15.12:0/2442750462 shutdown_connections 2024-03-17T23:52:35.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.125+0000 7f74bd000700 1 -- 172.21.15.12:0/2442750462 wait complete. 2024-03-17T23:52:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.127+0000 7f74bd000700 1 Processor -- start 2024-03-17T23:52:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.127+0000 7f74bd000700 1 -- start start 2024-03-17T23:52:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.128+0000 7f74bd000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74b8074cb0 0x7f74b819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:52:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.128+0000 7f74bd000700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74b8076280 0x7f74b819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:52:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.128+0000 7f74bd000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74b810a230 0x7f74b81a1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:52:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.128+0000 7f74bd000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f74b81100f0 con 0x7f74b810a230 2024-03-17T23:52:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.128+0000 7f74bd000700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f74b810ff70 con 0x7f74b8076280 2024-03-17T23:52:35.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.128+0000 7f74bd000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f74b8110270 con 0x7f74b8074cb0 2024-03-17T23:52:35.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.128+0000 7f74b5400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74b8076280 0x7f74b819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:52:35.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.128+0000 7f74b5400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74b8076280 0x7f74b819abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:57390/0 (socket says 172.21.15.12:57390) 2024-03-17T23:52:35.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.128+0000 7f74b5400700 1 -- 172.21.15.12:0/1320187708 learned_addr learned my addr 172.21.15.12:0/1320187708 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:52:35.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.128+0000 7f74b5e00700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74b8074cb0 0x7f74b819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:52:35.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.128+0000 7f74b6800700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74b810a230 0x7f74b81a1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:52:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.129+0000 7f74b5e00700 1 -- 172.21.15.12:0/1320187708 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74b8076280 msgr2=0x7f74b819abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:52:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.129+0000 7f74b5e00700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74b8076280 0x7f74b819abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.129+0000 7f74b5e00700 1 -- 172.21.15.12:0/1320187708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74b810a230 msgr2=0x7f74b81a1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:52:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.129+0000 7f74b5e00700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74b810a230 0x7f74b81a1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:52:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.129+0000 7f74b5e00700 1 -- 172.21.15.12:0/1320187708 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b81a22e0 con 0x7f74b8074cb0 2024-03-17T23:52:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.129+0000 7f74b5400700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74b8076280 0x7f74b819abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:52:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.129+0000 7f74b5e00700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74b8074cb0 0x7f74b819a680 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f74a8007fa0 tx=0x7f74a8004ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:52:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.129+0000 7f74aea00700 1 -- 172.21.15.12:0/1320187708 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a8021070 con 0x7f74b8074cb0 2024-03-17T23:52:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.129+0000 7f74aea00700 1 -- 172.21.15.12:0/1320187708 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74a8007b80 con 0x7f74b8074cb0 2024-03-17T23:52:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.129+0000 7f74aea00700 1 -- 172.21.15.12:0/1320187708 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a801d450 con 0x7f74b8074cb0 2024-03-17T23:52:35.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.129+0000 7f74bd000700 1 -- 172.21.15.12:0/1320187708 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b81a2570 con 0x7f74b8074cb0 2024-03-17T23:52:35.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.130+0000 7f74bd000700 1 -- 172.21.15.12:0/1320187708 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b81a2a20 con 0x7f74b8074cb0 2024-03-17T23:52:35.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.131+0000 7f74bd000700 1 -- 172.21.15.12:0/1320187708 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74b8068f90 con 0x7f74b8074cb0 2024-03-17T23:52:35.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.133+0000 7f74aea00700 1 -- 172.21.15.12:0/1320187708 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f74a8004000 con 0x7f74b8074cb0 2024-03-17T23:52:35.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.134+0000 7f74aea00700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f749c0a2890 0x7f749c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:52:35.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.134+0000 7f74aea00700 1 -- 172.21.15.12:0/1320187708 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f74a800f020 con 0x7f74b8074cb0 2024-03-17T23:52:35.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.134+0000 7f74b5400700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f749c0a2890 0x7f749c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:52:35.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.135+0000 7f74b5400700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f749c0a2890 0x7f749c0a4d50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f74a4011550 tx=0x7f74a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:52:35.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.139+0000 7f74aea00700 1 -- 172.21.15.12:0/1320187708 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f74a802a020 con 0x7f74b8074cb0 2024-03-17T23:52:35.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:52:35.377+0000 7f74bd000700 1 -- 172.21.15.12:0/1320187708 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f74b8062710 con 0x7f749c0a2890 2024-03-17T23:52:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:35 smithi012 ceph-mon[31541]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-17T23:52:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:52:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:52:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:35.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:35 smithi079 ceph-mon[36241]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-17T23:52:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:52:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:52:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:35 smithi191 ceph-mon[36708]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-17T23:52:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:52:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:52:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:36.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:36 smithi012 ceph-mon[31541]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:52:36.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:36 smithi079 ceph-mon[36241]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:52:36.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:36 smithi191 ceph-mon[36708]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:52:37.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:37 smithi012 ceph-mon[31541]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-17T23:52:37.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:37 smithi079 ceph-mon[36241]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-17T23:52:37.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:37 smithi191 ceph-mon[36708]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-17T23:52:39.552 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:39 smithi012 ceph-mon[31541]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-17T23:52:39.552 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:39 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1826026555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8a4b561-6262-4a05-855f-086c7d39a955"}]: dispatch 2024-03-17T23:52:39.552 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:39 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1826026555' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8a4b561-6262-4a05-855f-086c7d39a955"}]': finished 2024-03-17T23:52:39.552 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:39 smithi012 ceph-mon[31541]: osdmap e64: 10 total, 9 up, 10 in 2024-03-17T23:52:39.553 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:52:39.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:39 smithi079 ceph-mon[36241]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-17T23:52:39.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:39 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1826026555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8a4b561-6262-4a05-855f-086c7d39a955"}]: dispatch 2024-03-17T23:52:39.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:39 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1826026555' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8a4b561-6262-4a05-855f-086c7d39a955"}]': finished 2024-03-17T23:52:39.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:39 smithi079 ceph-mon[36241]: osdmap e64: 10 total, 9 up, 10 in 2024-03-17T23:52:39.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:52:39.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:39 smithi191 ceph-mon[36708]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-17T23:52:39.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:39 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1826026555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8a4b561-6262-4a05-855f-086c7d39a955"}]: dispatch 2024-03-17T23:52:39.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:39 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1826026555' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8a4b561-6262-4a05-855f-086c7d39a955"}]': finished 2024-03-17T23:52:39.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:39 smithi191 ceph-mon[36708]: osdmap e64: 10 total, 9 up, 10 in 2024-03-17T23:52:39.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:52:40.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:40 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2825219192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:52:40.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:40 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2825219192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:52:40.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:40 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2825219192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:52:41.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:41 smithi079 ceph-mon[36241]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-17T23:52:41.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:41 smithi191 ceph-mon[36708]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-17T23:52:41.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:41 smithi012 ceph-mon[31541]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-17T23:52:43.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:43 smithi079 ceph-mon[36241]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-17T23:52:43.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:43 smithi191 ceph-mon[36708]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-17T23:52:43.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:43 smithi012 ceph-mon[31541]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-17T23:52:45.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:45 smithi079 ceph-mon[36241]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:45.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:45 smithi191 ceph-mon[36708]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:45.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:45 smithi012 ceph-mon[31541]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:47.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:47 smithi079 ceph-mon[36241]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:47.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:47 smithi191 ceph-mon[36708]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:47.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:47 smithi012 ceph-mon[31541]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:49.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:49 smithi191 ceph-mon[36708]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:49.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:49 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-17T23:52:49.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:49 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:49.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:49 smithi191 ceph-mon[36708]: Deploying daemon osd.9 on smithi012 2024-03-17T23:52:49.879 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:49 smithi012 ceph-mon[31541]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:49.879 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:49 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-17T23:52:49.879 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:49 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:49.879 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:49 smithi012 ceph-mon[31541]: Deploying daemon osd.9 on smithi012 2024-03-17T23:52:50.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:49 smithi079 ceph-mon[36241]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:50.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:49 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-17T23:52:50.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:49 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:52:50.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:49 smithi079 ceph-mon[36241]: Deploying daemon osd.9 on smithi012 2024-03-17T23:52:51.828 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:51 smithi012 ceph-mon[31541]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:51.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:51 smithi191 ceph-mon[36708]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:52.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:51 smithi079 ceph-mon[36241]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:53.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:53 smithi191 ceph-mon[36708]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:53.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:53 smithi012 ceph-mon[31541]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:54.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:53 smithi079 ceph-mon[36241]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:55 smithi191 ceph-mon[36708]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:55.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:55 smithi012 ceph-mon[31541]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:56.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:55 smithi079 ceph-mon[36241]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:57 smithi191 ceph-mon[36708]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:57.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:57 smithi012 ceph-mon[31541]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:58.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:57 smithi079 ceph-mon[36241]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:52:58.976 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:58 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:58.977 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:58 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:58.977 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:58 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:52:59.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:58 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:59.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:58 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:59.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:58 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:52:59.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:58 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:59.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:58 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:52:59.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:58 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:52:59.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:52:59 smithi012 ceph-mon[31541]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:53:00.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:52:59 smithi079 ceph-mon[36241]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:53:00.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:52:59 smithi191 ceph-mon[36708]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:53:01.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:01 smithi012 ceph-mon[31541]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:53:01.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:01 smithi012 ceph-mon[31541]: from='osd.9 [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-17T23:53:01.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:01 smithi012 ceph-mon[31541]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-17T23:53:01.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:01 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:01.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:01 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:02.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:01 smithi079 ceph-mon[36241]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:53:02.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:01 smithi079 ceph-mon[36241]: from='osd.9 [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-17T23:53:02.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:01 smithi079 ceph-mon[36241]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-17T23:53:02.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:01 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:02.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:01 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:01 smithi191 ceph-mon[36708]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:53:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:01 smithi191 ceph-mon[36708]: from='osd.9 [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-17T23:53:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:01 smithi191 ceph-mon[36708]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-17T23:53:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:01 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:01 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:02.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:02.519+0000 7f74aea00700 1 -- 172.21.15.12:0/1320187708 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f74b8062710 con 0x7f749c0a2890 2024-03-17T23:53:02.521 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 9 on host 'smithi012' 2024-03-17T23:53:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:02.525+0000 7f74bd000700 1 -- 172.21.15.12:0/1320187708 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f749c0a2890 msgr2=0x7f749c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:02.525+0000 7f74bd000700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f749c0a2890 0x7f749c0a4d50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f74a4011550 tx=0x7f74a4009260 comp rx=0 tx=0).stop 2024-03-17T23:53:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:02.525+0000 7f74bd000700 1 -- 172.21.15.12:0/1320187708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74b8074cb0 msgr2=0x7f74b819a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:02.525+0000 7f74bd000700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74b8074cb0 0x7f74b819a680 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f74a8007fa0 tx=0x7f74a8004ce0 comp rx=0 tx=0).stop 2024-03-17T23:53:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:02.525+0000 7f74bd000700 1 -- 172.21.15.12:0/1320187708 shutdown_connections 2024-03-17T23:53:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:02.525+0000 7f74bd000700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f749c0a2890 0x7f749c0a4d50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:02.525+0000 7f74bd000700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74b810a230 0x7f74b81a1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:02.525+0000 7f74bd000700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74b8074cb0 0x7f74b819a680 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:02.525+0000 7f74bd000700 1 --2- 172.21.15.12:0/1320187708 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74b8076280 0x7f74b819abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:02.525+0000 7f74bd000700 1 -- 172.21.15.12:0/1320187708 >> 172.21.15.12:0/1320187708 conn(0x7f74b80fd8d0 msgr2=0x7f74b8106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:53:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:02.525+0000 7f74bd000700 1 -- 172.21.15.12:0/1320187708 shutdown_connections 2024-03-17T23:53:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:02.525+0000 7f74bd000700 1 -- 172.21.15.12:0/1320187708 wait complete. 2024-03-17T23:53:02.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:02 smithi012 ceph-mon[31541]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-17T23:53:02.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:02 smithi012 ceph-mon[31541]: from='osd.9 [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:53:02.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:02 smithi012 ceph-mon[31541]: osdmap e65: 10 total, 9 up, 10 in 2024-03-17T23:53:02.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:02 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:02.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:02 smithi012 ceph-mon[31541]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:53:02.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:02 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:02.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:02 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:02.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:02 smithi012 ceph-mon[31541]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-17T23:53:02.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:02 smithi012 ceph-mon[31541]: osdmap e66: 10 total, 9 up, 10 in 2024-03-17T23:53:02.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:02 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:03.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:02 smithi079 ceph-mon[36241]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-17T23:53:03.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:02 smithi079 ceph-mon[36241]: from='osd.9 [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:53:03.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:02 smithi079 ceph-mon[36241]: osdmap e65: 10 total, 9 up, 10 in 2024-03-17T23:53:03.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:02 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:03.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:02 smithi079 ceph-mon[36241]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:53:03.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:02 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:03.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:02 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:03.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:02 smithi079 ceph-mon[36241]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-17T23:53:03.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:02 smithi079 ceph-mon[36241]: osdmap e66: 10 total, 9 up, 10 in 2024-03-17T23:53:03.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:02 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:03.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:02 smithi191 ceph-mon[36708]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-17T23:53:03.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:02 smithi191 ceph-mon[36708]: from='osd.9 [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:53:03.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:02 smithi191 ceph-mon[36708]: osdmap e65: 10 total, 9 up, 10 in 2024-03-17T23:53:03.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:02 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:03.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:02 smithi191 ceph-mon[36708]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-17T23:53:03.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:02 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:03.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:02 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:03.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:02 smithi191 ceph-mon[36708]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-17T23:53:03.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:02 smithi191 ceph-mon[36708]: osdmap e66: 10 total, 9 up, 10 in 2024-03-17T23:53:03.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:02 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:03.375 DEBUG:teuthology.orchestra.run.smithi012:osd.9> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.9.service 2024-03-17T23:53:03.381 INFO:tasks.cephadm:Deploying osd.10 on smithi079 with /dev/vg_nvme/lv_1... 2024-03-17T23:53:03.381 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-17T23:53:03.628 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.b/config 2024-03-17T23:53:03.668 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 17 23:53:03 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-17T23:53:03.458+0000 7f3bc4000700 -1 osd.9 0 waiting for initial osdmap 2024-03-17T23:53:03.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 17 23:53:03 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-17T23:53:03.493+0000 7f3bb9e00700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:53:04.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:03 smithi079 ceph-mon[36241]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:53:04.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:03 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:04.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:03 smithi079 ceph-mon[36241]: from='osd.9 ' entity='osd.9' 2024-03-17T23:53:04.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:03 smithi191 ceph-mon[36708]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:53:04.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:03 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:04.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:03 smithi191 ceph-mon[36708]: from='osd.9 ' entity='osd.9' 2024-03-17T23:53:04.098 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:03 smithi012 ceph-mon[31541]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:53:04.098 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:03 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:04.098 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:03 smithi012 ceph-mon[31541]: from='osd.9 ' entity='osd.9' 2024-03-17T23:53:05.033 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:04 smithi079 ceph-mon[36241]: purged_snaps scrub starts 2024-03-17T23:53:05.033 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:04 smithi079 ceph-mon[36241]: purged_snaps scrub ok 2024-03-17T23:53:05.033 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:04 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:05.033 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:04 smithi079 ceph-mon[36241]: osd.9 [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796] boot 2024-03-17T23:53:05.034 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:04 smithi079 ceph-mon[36241]: osdmap e67: 10 total, 10 up, 10 in 2024-03-17T23:53:05.034 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:04 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:05.069 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:04 smithi012 ceph-mon[31541]: purged_snaps scrub starts 2024-03-17T23:53:05.069 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:04 smithi012 ceph-mon[31541]: purged_snaps scrub ok 2024-03-17T23:53:05.069 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:04 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:05.069 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:04 smithi012 ceph-mon[31541]: osd.9 [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796] boot 2024-03-17T23:53:05.069 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:04 smithi012 ceph-mon[31541]: osdmap e67: 10 total, 10 up, 10 in 2024-03-17T23:53:05.069 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:04 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:05.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:04 smithi191 ceph-mon[36708]: purged_snaps scrub starts 2024-03-17T23:53:05.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:04 smithi191 ceph-mon[36708]: purged_snaps scrub ok 2024-03-17T23:53:05.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:04 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:05.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:04 smithi191 ceph-mon[36708]: osd.9 [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796] boot 2024-03-17T23:53:05.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:04 smithi191 ceph-mon[36708]: osdmap e67: 10 total, 10 up, 10 in 2024-03-17T23:53:05.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:04 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-17T23:53:05.723 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-17T23:53:05.723 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-17T23:53:05.723 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-03-17T23:53:05.723 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-03-17T23:53:05.723 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111696 s, 939 MB/s 2024-03-17T23:53:05.723 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-03-17T23:53:06.039 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:05 smithi079 ceph-mon[36241]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:53:06.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:05 smithi191 ceph-mon[36708]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:53:06.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:05 smithi012 ceph-mon[31541]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-17T23:53:06.335 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi079:vg_nvme/lv_1 2024-03-17T23:53:06.579 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.b/config 2024-03-17T23:53:06.858 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:06 smithi079 ceph-mon[36241]: osdmap e68: 10 total, 10 up, 10 in 2024-03-17T23:53:06.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:06 smithi012 ceph-mon[31541]: osdmap e68: 10 total, 10 up, 10 in 2024-03-17T23:53:07.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:06 smithi191 ceph-mon[36708]: osdmap e68: 10 total, 10 up, 10 in 2024-03-17T23:53:08.076 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:07 smithi079 ceph-mon[36241]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:08.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.084+0000 7f64f331a700 1 -- 172.21.15.79:0/3253107942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64ec102e10 msgr2=0x7f64ec1096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:08.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.084+0000 7f64f331a700 1 --2- 172.21.15.79:0/3253107942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64ec102e10 0x7f64ec1096c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f64e400b0d0 tx=0x7f64e400b3e0 comp rx=0 tx=0).stop 2024-03-17T23:53:08.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.084+0000 7f64f331a700 1 -- 172.21.15.79:0/3253107942 shutdown_connections 2024-03-17T23:53:08.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.084+0000 7f64f331a700 1 --2- 172.21.15.79:0/3253107942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64ec102e10 0x7f64ec1096c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:08.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.084+0000 7f64f331a700 1 --2- 172.21.15.79:0/3253107942 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64ec101270 0x7f64ec101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:08.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.084+0000 7f64f331a700 1 --2- 172.21.15.79:0/3253107942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64ec102470 0x7f64ec1028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:08.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.084+0000 7f64f331a700 1 -- 172.21.15.79:0/3253107942 >> 172.21.15.79:0/3253107942 conn(0x7f64ec0fc830 msgr2=0x7f64ec0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:53:08.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.084+0000 7f64f331a700 1 -- 172.21.15.79:0/3253107942 shutdown_connections 2024-03-17T23:53:08.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.084+0000 7f64f331a700 1 -- 172.21.15.79:0/3253107942 wait complete. 2024-03-17T23:53:08.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.085+0000 7f64f331a700 1 Processor -- start 2024-03-17T23:53:08.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.085+0000 7f64f331a700 1 -- start start 2024-03-17T23:53:08.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.085+0000 7f64f331a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64ec101270 0x7f64ec19a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:53:08.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.085+0000 7f64f331a700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64ec102470 0x7f64ec19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:53:08.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.085+0000 7f64f331a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64ec102e10 0x7f64ec1a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:53:08.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.085+0000 7f64f331a700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec10cee0 con 0x7f64ec102470 2024-03-17T23:53:08.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.085+0000 7f64f331a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec10cd60 con 0x7f64ec102e10 2024-03-17T23:53:08.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.085+0000 7f64f331a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec10d060 con 0x7f64ec101270 2024-03-17T23:53:08.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.085+0000 7f64f18b7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64ec102e10 0x7f64ec1a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:53:08.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.085+0000 7f64f18b7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64ec102e10 0x7f64ec1a1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50182/0 (socket says 172.21.15.79:50182) 2024-03-17T23:53:08.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.085+0000 7f64f18b7700 1 -- 172.21.15.79:0/678715716 learned_addr learned my addr 172.21.15.79:0/678715716 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-03-17T23:53:08.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.085+0000 7f64f10b6700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64ec101270 0x7f64ec19a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:53:08.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.086+0000 7f64f18b7700 1 -- 172.21.15.79:0/678715716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64ec101270 msgr2=0x7f64ec19a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:08.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.086+0000 7f64f18b7700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64ec101270 0x7f64ec19a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:08.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.086+0000 7f64f18b7700 1 -- 172.21.15.79:0/678715716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64ec102470 msgr2=0x7f64ec19ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:08.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.086+0000 7f64f18b7700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64ec102470 0x7f64ec19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:08.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.086+0000 7f64f18b7700 1 -- 172.21.15.79:0/678715716 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ec1a2450 con 0x7f64ec102e10 2024-03-17T23:53:08.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.086+0000 7f64f10b6700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64ec101270 0x7f64ec19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:53:08.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.086+0000 7f64f18b7700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64ec102e10 0x7f64ec1a1db0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f64e40062a0 tx=0x7f64e4007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:53:08.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.086+0000 7f64e27fc700 1 -- 172.21.15.79:0/678715716 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4003c00 con 0x7f64ec102e10 2024-03-17T23:53:08.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.086+0000 7f64e27fc700 1 -- 172.21.15.79:0/678715716 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64e4003da0 con 0x7f64ec102e10 2024-03-17T23:53:08.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.086+0000 7f64e27fc700 1 -- 172.21.15.79:0/678715716 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4005560 con 0x7f64ec102e10 2024-03-17T23:53:08.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.086+0000 7f64f331a700 1 -- 172.21.15.79:0/678715716 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64ec1a26e0 con 0x7f64ec102e10 2024-03-17T23:53:08.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.086+0000 7f64f331a700 1 -- 172.21.15.79:0/678715716 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64ec1a2c40 con 0x7f64ec102e10 2024-03-17T23:53:08.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:07 smithi191 ceph-mon[36708]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:08.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.087+0000 7f64f331a700 1 -- 172.21.15.79:0/678715716 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64ec068f90 con 0x7f64ec102e10 2024-03-17T23:53:08.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.088+0000 7f64e27fc700 1 -- 172.21.15.79:0/678715716 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f64e4005700 con 0x7f64ec102e10 2024-03-17T23:53:08.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.089+0000 7f64e27fc700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f64d80a2910 0x7f64d80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:53:08.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.089+0000 7f64e27fc700 1 -- 172.21.15.79:0/678715716 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f64e40cf3a0 con 0x7f64ec102e10 2024-03-17T23:53:08.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.089+0000 7f64f10b6700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f64d80a2910 0x7f64d80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:53:08.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.090+0000 7f64f10b6700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f64d80a2910 0x7f64d80a4dd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f64ec1022d0 tx=0x7f64e8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:53:08.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.093+0000 7f64e27fc700 1 -- 172.21.15.79:0/678715716 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f64e4009cb0 con 0x7f64ec102e10 2024-03-17T23:53:08.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:07 smithi012 ceph-mon[31541]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:08.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:08.300+0000 7f64f331a700 1 -- 172.21.15.79:0/678715716 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f64ec0fea00 con 0x7f64d80a2910 2024-03-17T23:53:09.076 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:53:09.077 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:53:09.077 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:09.077 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:09.077 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:09.077 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:09.077 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:09.077 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:09.077 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:09.077 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:09.077 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:09.077 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:09.077 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:08 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:09.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:53:09.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:53:09.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:09.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:09.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:09.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:09.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:09.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:09.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:09.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:09.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:09.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:09.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:08 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:09.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:53:09.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:53:09.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:09.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:09.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:09.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:09.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:09.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:09.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:09.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:09.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:09.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:09.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:08 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:10.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:09 smithi079 ceph-mon[36241]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:10.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:09 smithi079 ceph-mon[36241]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:53:10.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:09 smithi079 ceph-mon[36241]: Detected new or changed devices on smithi012 2024-03-17T23:53:10.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:09 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi012 to 4273M 2024-03-17T23:53:10.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:09 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:10.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:09 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:10.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:09 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:10.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:09 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:10.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:09 smithi191 ceph-mon[36708]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:10.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:09 smithi191 ceph-mon[36708]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:53:10.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:09 smithi191 ceph-mon[36708]: Detected new or changed devices on smithi012 2024-03-17T23:53:10.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:09 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi012 to 4273M 2024-03-17T23:53:10.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:09 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:10.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:09 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:10.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:09 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:10.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:09 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:09 smithi012 ceph-mon[31541]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:09 smithi012 ceph-mon[31541]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:53:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:09 smithi012 ceph-mon[31541]: Detected new or changed devices on smithi012 2024-03-17T23:53:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:09 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi012 to 4273M 2024-03-17T23:53:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:12.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:11 smithi191 ceph-mon[36708]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:12.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:11 smithi191 ceph-mon[36708]: from='client.? 172.21.15.79:0/1313108527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d562fe1-e7a4-41f6-81e9-95f8745539bd"}]: dispatch 2024-03-17T23:53:12.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:11 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d562fe1-e7a4-41f6-81e9-95f8745539bd"}]: dispatch 2024-03-17T23:53:12.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:11 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d562fe1-e7a4-41f6-81e9-95f8745539bd"}]': finished 2024-03-17T23:53:12.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:11 smithi191 ceph-mon[36708]: osdmap e69: 11 total, 10 up, 11 in 2024-03-17T23:53:12.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-17T23:53:12.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:11 smithi012 ceph-mon[31541]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:12.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:11 smithi012 ceph-mon[31541]: from='client.? 172.21.15.79:0/1313108527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d562fe1-e7a4-41f6-81e9-95f8745539bd"}]: dispatch 2024-03-17T23:53:12.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:11 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d562fe1-e7a4-41f6-81e9-95f8745539bd"}]: dispatch 2024-03-17T23:53:12.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:11 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d562fe1-e7a4-41f6-81e9-95f8745539bd"}]': finished 2024-03-17T23:53:12.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:11 smithi012 ceph-mon[31541]: osdmap e69: 11 total, 10 up, 11 in 2024-03-17T23:53:12.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-17T23:53:12.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:11 smithi079 ceph-mon[36241]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:12.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:11 smithi079 ceph-mon[36241]: from='client.? 172.21.15.79:0/1313108527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d562fe1-e7a4-41f6-81e9-95f8745539bd"}]: dispatch 2024-03-17T23:53:12.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:11 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d562fe1-e7a4-41f6-81e9-95f8745539bd"}]: dispatch 2024-03-17T23:53:12.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:11 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d562fe1-e7a4-41f6-81e9-95f8745539bd"}]': finished 2024-03-17T23:53:12.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:11 smithi079 ceph-mon[36241]: osdmap e69: 11 total, 10 up, 11 in 2024-03-17T23:53:12.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-17T23:53:13.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:12 smithi191 ceph-mon[36708]: from='client.? 172.21.15.79:0/13854790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:53:13.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:12 smithi012 ceph-mon[31541]: from='client.? 172.21.15.79:0/13854790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:53:13.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:12 smithi079 ceph-mon[36241]: from='client.? 172.21.15.79:0/13854790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:53:14.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:13 smithi012 ceph-mon[31541]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:14.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:13 smithi079 ceph-mon[36241]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:14.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:13 smithi191 ceph-mon[36708]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:16.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:15 smithi012 ceph-mon[31541]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:16.244 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:15 smithi079 ceph-mon[36241]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:16.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:15 smithi191 ceph-mon[36708]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:18.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:17 smithi012 ceph-mon[31541]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:18.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:17 smithi079 ceph-mon[36241]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:18.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:17 smithi191 ceph-mon[36708]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:19.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:18 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-17T23:53:19.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:18 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:19.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:18 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-17T23:53:19.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:18 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:19.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:18 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-17T23:53:19.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:18 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:19 smithi079 ceph-mon[36241]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:20.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:19 smithi079 ceph-mon[36241]: Deploying daemon osd.10 on smithi079 2024-03-17T23:53:20.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:19 smithi012 ceph-mon[31541]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:20.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:19 smithi012 ceph-mon[31541]: Deploying daemon osd.10 on smithi079 2024-03-17T23:53:20.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:19 smithi191 ceph-mon[36708]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:20.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:19 smithi191 ceph-mon[36708]: Deploying daemon osd.10 on smithi079 2024-03-17T23:53:22.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:21 smithi012 ceph-mon[31541]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:22.202 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:21 smithi079 ceph-mon[36241]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:22.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:21 smithi191 ceph-mon[36708]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:24.136 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:23 smithi079 ceph-mon[36241]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:24.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:23 smithi012 ceph-mon[31541]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:24.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:23 smithi191 ceph-mon[36708]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:25.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:25 smithi012 ceph-mon[31541]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:25.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:25.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:25.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:26.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:25 smithi079 ceph-mon[36241]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:26.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:26.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:26.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:26.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:25 smithi191 ceph-mon[36708]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:26.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:26.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:26.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:27.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:27 smithi079 ceph-mon[36241]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:27.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:27 smithi012 ceph-mon[31541]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:28.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:27 smithi191 ceph-mon[36708]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:28.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:28.540+0000 7f64e27fc700 1 -- 172.21.15.79:0/678715716 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f64ec0fea00 con 0x7f64d80a2910 2024-03-17T23:53:28.540 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 10 on host 'smithi079' 2024-03-17T23:53:28.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:28.542+0000 7f64f331a700 1 -- 172.21.15.79:0/678715716 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f64d80a2910 msgr2=0x7f64d80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:28.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:28.542+0000 7f64f331a700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f64d80a2910 0x7f64d80a4dd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f64ec1022d0 tx=0x7f64e8008040 comp rx=0 tx=0).stop 2024-03-17T23:53:28.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:28.542+0000 7f64f331a700 1 -- 172.21.15.79:0/678715716 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64ec102e10 msgr2=0x7f64ec1a1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:28.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:28.542+0000 7f64f331a700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64ec102e10 0x7f64ec1a1db0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f64e40062a0 tx=0x7f64e4007e40 comp rx=0 tx=0).stop 2024-03-17T23:53:28.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:28.542+0000 7f64f331a700 1 -- 172.21.15.79:0/678715716 shutdown_connections 2024-03-17T23:53:28.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:28.542+0000 7f64f331a700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f64d80a2910 0x7f64d80a4dd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:28.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:28.542+0000 7f64f331a700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64ec102e10 0x7f64ec1a1db0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:28.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:28.542+0000 7f64f331a700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64ec101270 0x7f64ec19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:28.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:28.542+0000 7f64f331a700 1 --2- 172.21.15.79:0/678715716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64ec102470 0x7f64ec19ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:28.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:28.542+0000 7f64f331a700 1 -- 172.21.15.79:0/678715716 >> 172.21.15.79:0/678715716 conn(0x7f64ec0fc830 msgr2=0x7f64ec0fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:53:28.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:28.542+0000 7f64f331a700 1 -- 172.21.15.79:0/678715716 shutdown_connections 2024-03-17T23:53:28.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-17T23:53:28.542+0000 7f64f331a700 1 -- 172.21.15.79:0/678715716 wait complete. 2024-03-17T23:53:29.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:29.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:29.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:28 smithi191 ceph-mon[36708]: from='osd.10 [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-17T23:53:29.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:28 smithi191 ceph-mon[36708]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-17T23:53:29.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:29.153 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.153 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.153 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:29.153 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:29.153 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.153 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:28 smithi079 ceph-mon[36241]: from='osd.10 [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-17T23:53:29.153 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:28 smithi079 ceph-mon[36241]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-17T23:53:29.153 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.153 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.153 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:29.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:29.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:29.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:28 smithi012 ceph-mon[31541]: from='osd.10 [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-17T23:53:29.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:28 smithi012 ceph-mon[31541]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-17T23:53:29.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:29.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:29.178 DEBUG:teuthology.orchestra.run.smithi079:osd.10> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.10.service 2024-03-17T23:53:29.180 INFO:tasks.cephadm:Deploying osd.11 on smithi191 with /dev/vg_nvme/lv_1... 2024-03-17T23:53:29.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-17T23:53:29.426 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-17T23:53:30.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:29 smithi012 ceph-mon[31541]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:30.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:29 smithi012 ceph-mon[31541]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-17T23:53:30.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:29 smithi012 ceph-mon[31541]: from='osd.10 [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:53:30.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:29 smithi012 ceph-mon[31541]: osdmap e70: 11 total, 10 up, 11 in 2024-03-17T23:53:30.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:29 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-17T23:53:30.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:29 smithi012 ceph-mon[31541]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:53:30.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:29 smithi079 ceph-mon[36241]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:30.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:29 smithi079 ceph-mon[36241]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-17T23:53:30.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:29 smithi079 ceph-mon[36241]: from='osd.10 [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:53:30.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:29 smithi079 ceph-mon[36241]: osdmap e70: 11 total, 10 up, 11 in 2024-03-17T23:53:30.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:29 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-17T23:53:30.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:29 smithi079 ceph-mon[36241]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:53:30.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 17 23:53:29 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-17T23:53:29.843+0000 7fec6daac700 -1 osd.10 0 waiting for initial osdmap 2024-03-17T23:53:30.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 17 23:53:29 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-17T23:53:29.855+0000 7fec6989f700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:53:30.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:29 smithi191 ceph-mon[36708]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:30.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:29 smithi191 ceph-mon[36708]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-17T23:53:30.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:29 smithi191 ceph-mon[36708]: from='osd.10 [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:53:30.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:29 smithi191 ceph-mon[36708]: osdmap e70: 11 total, 10 up, 11 in 2024-03-17T23:53:30.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:29 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-17T23:53:30.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:29 smithi191 ceph-mon[36708]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-17T23:53:30.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:30 smithi191 ceph-mon[36708]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-17T23:53:30.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:30 smithi191 ceph-mon[36708]: osdmap e71: 11 total, 10 up, 11 in 2024-03-17T23:53:30.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:30 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-17T23:53:30.939 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:30 smithi079 ceph-mon[36241]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-17T23:53:30.939 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:30 smithi079 ceph-mon[36241]: osdmap e71: 11 total, 10 up, 11 in 2024-03-17T23:53:30.940 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:30 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-17T23:53:31.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:30 smithi012 ceph-mon[31541]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-17T23:53:31.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:30 smithi012 ceph-mon[31541]: osdmap e71: 11 total, 10 up, 11 in 2024-03-17T23:53:31.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:30 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-17T23:53:31.581 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-17T23:53:31.581 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-17T23:53:31.581 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-03-17T23:53:31.581 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-03-17T23:53:31.581 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113743 s, 922 MB/s 2024-03-17T23:53:31.581 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-03-17T23:53:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:31 smithi012 ceph-mon[31541]: purged_snaps scrub starts 2024-03-17T23:53:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:31 smithi012 ceph-mon[31541]: purged_snaps scrub ok 2024-03-17T23:53:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:31 smithi012 ceph-mon[31541]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:31 smithi012 ceph-mon[31541]: osd.10 [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359] boot 2024-03-17T23:53:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:31 smithi012 ceph-mon[31541]: osdmap e72: 11 total, 11 up, 11 in 2024-03-17T23:53:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-17T23:53:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:31 smithi012 ceph-mon[31541]: osdmap e73: 11 total, 11 up, 11 in 2024-03-17T23:53:32.203 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:31 smithi079 ceph-mon[36241]: purged_snaps scrub starts 2024-03-17T23:53:32.203 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:31 smithi079 ceph-mon[36241]: purged_snaps scrub ok 2024-03-17T23:53:32.203 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:31 smithi079 ceph-mon[36241]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:32.203 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:31 smithi079 ceph-mon[36241]: osd.10 [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359] boot 2024-03-17T23:53:32.203 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:31 smithi079 ceph-mon[36241]: osdmap e72: 11 total, 11 up, 11 in 2024-03-17T23:53:32.203 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-17T23:53:32.203 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:31 smithi079 ceph-mon[36241]: osdmap e73: 11 total, 11 up, 11 in 2024-03-17T23:53:32.205 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:31 smithi191 ceph-mon[36708]: purged_snaps scrub starts 2024-03-17T23:53:32.205 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:31 smithi191 ceph-mon[36708]: purged_snaps scrub ok 2024-03-17T23:53:32.205 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:31 smithi191 ceph-mon[36708]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-17T23:53:32.206 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:31 smithi191 ceph-mon[36708]: osd.10 [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359] boot 2024-03-17T23:53:32.206 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:31 smithi191 ceph-mon[36708]: osdmap e72: 11 total, 11 up, 11 in 2024-03-17T23:53:32.206 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-17T23:53:32.206 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:31 smithi191 ceph-mon[36708]: osdmap e73: 11 total, 11 up, 11 in 2024-03-17T23:53:32.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi191:vg_nvme/lv_1 2024-03-17T23:53:32.478 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-17T23:53:33.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:33.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: Detected new or changed devices on smithi079 2024-03-17T23:53:33.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:33.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:33.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:33.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:33.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:33.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:33.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi079 to 4270M 2024-03-17T23:53:33.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:33.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:33.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:33.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:33.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:33 smithi079 ceph-mon[36241]: osdmap e74: 11 total, 11 up, 11 in 2024-03-17T23:53:33.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:33.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: Detected new or changed devices on smithi079 2024-03-17T23:53:33.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:33.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:33.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:33.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:33.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:33.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:33.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi079 to 4270M 2024-03-17T23:53:33.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:33.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:33.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:33.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:33.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:33 smithi191 ceph-mon[36708]: osdmap e74: 11 total, 11 up, 11 in 2024-03-17T23:53:33.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:33.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: Detected new or changed devices on smithi079 2024-03-17T23:53:33.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:33.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:33.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:33.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:33.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:33.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:33.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi079 to 4270M 2024-03-17T23:53:33.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:33.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:33.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:33.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:33.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:33 smithi012 ceph-mon[31541]: osdmap e74: 11 total, 11 up, 11 in 2024-03-17T23:53:34.179 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.176+0000 7f08c754f700 1 -- 172.21.15.191:0/4127107041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08c0103ea0 msgr2=0x7f08c010a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:34.179 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.176+0000 7f08c754f700 1 --2- 172.21.15.191:0/4127107041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08c0103ea0 0x7f08c010a750 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f08b800b0a0 tx=0x7f08b800b3b0 comp rx=0 tx=0).stop 2024-03-17T23:53:34.179 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 -- 172.21.15.191:0/4127107041 shutdown_connections 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 --2- 172.21.15.191:0/4127107041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08c0103ea0 0x7f08c010a750 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 --2- 172.21.15.191:0/4127107041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08c0102300 0x7f08c0102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 --2- 172.21.15.191:0/4127107041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08c0103500 0x7f08c0103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 -- 172.21.15.191:0/4127107041 >> 172.21.15.191:0/4127107041 conn(0x7f08c00fd8e0 msgr2=0x7f08c00ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 -- 172.21.15.191:0/4127107041 shutdown_connections 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 -- 172.21.15.191:0/4127107041 wait complete. 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 Processor -- start 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 -- start start 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08c0102300 0x7f08c0076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08c0103500 0x7f08c00746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08c0103ea0 0x7f08c0074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f08c010dfe0 con 0x7f08c0103500 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f08c010de60 con 0x7f08c0102300 2024-03-17T23:53:34.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.177+0000 7f08c754f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f08c010e160 con 0x7f08c0103ea0 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08c5aec700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08c0103ea0 0x7f08c0074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08c52eb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08c0102300 0x7f08c0076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08c5aec700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08c0103ea0 0x7f08c0074c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:44088/0 (socket says 172.21.15.191:44088) 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08c5aec700 1 -- 172.21.15.191:0/3013676826 learned_addr learned my addr 172.21.15.191:0/3013676826 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08c5aec700 1 -- 172.21.15.191:0/3013676826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08c0102300 msgr2=0x7f08c0076070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08c5aec700 1 --2- 172.21.15.191:0/3013676826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08c0102300 0x7f08c0076070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08c5aec700 1 -- 172.21.15.191:0/3013676826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08c0103500 msgr2=0x7f08c00746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08c5aec700 1 --2- 172.21.15.191:0/3013676826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08c0103500 0x7f08c00746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08c5aec700 1 -- 172.21.15.191:0/3013676826 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c00754b0 con 0x7f08c0103ea0 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08c5aec700 1 --2- 172.21.15.191:0/3013676826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08c0103ea0 0x7f08c0074c00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f08b800b070 tx=0x7f08b800be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08b67fc700 1 -- 172.21.15.191:0/3013676826 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b8021020 con 0x7f08c0103ea0 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08b67fc700 1 -- 172.21.15.191:0/3013676826 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08b8005520 con 0x7f08c0103ea0 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08c754f700 1 -- 172.21.15.191:0/3013676826 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c01a2760 con 0x7f08c0103ea0 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08b67fc700 1 -- 172.21.15.191:0/3013676826 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b801ba00 con 0x7f08c0103ea0 2024-03-17T23:53:34.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.178+0000 7f08c754f700 1 -- 172.21.15.191:0/3013676826 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c01a2c40 con 0x7f08c0103ea0 2024-03-17T23:53:34.182 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.179+0000 7f08c754f700 1 -- 172.21.15.191:0/3013676826 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08c0068f90 con 0x7f08c0103ea0 2024-03-17T23:53:34.182 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.180+0000 7f08b67fc700 1 -- 172.21.15.191:0/3013676826 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f08b8024020 con 0x7f08c0103ea0 2024-03-17T23:53:34.182 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.181+0000 7f08b67fc700 1 --2- 172.21.15.191:0/3013676826 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f08ac0a2890 0x7f08ac0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:53:34.182 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.181+0000 7f08b67fc700 1 -- 172.21.15.191:0/3013676826 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f08b800f020 con 0x7f08c0103ea0 2024-03-17T23:53:34.183 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.182+0000 7f08c52eb700 1 --2- 172.21.15.191:0/3013676826 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f08ac0a2890 0x7f08ac0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:53:34.184 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.183+0000 7f08c52eb700 1 --2- 172.21.15.191:0/3013676826 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f08ac0a2890 0x7f08ac0a4d50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f08c0103360 tx=0x7f08bc009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:53:34.186 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.186+0000 7f08b67fc700 1 -- 172.21.15.191:0/3013676826 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f08b809cd50 con 0x7f08c0103ea0 2024-03-17T23:53:34.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:34.392+0000 7f08c754f700 1 -- 172.21.15.191:0/3013676826 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f08c0062710 con 0x7f08ac0a2890 2024-03-17T23:53:34.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:34 smithi012 ceph-mon[31541]: osdmap e75: 11 total, 11 up, 11 in 2024-03-17T23:53:34.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:34 smithi079 ceph-mon[36241]: osdmap e75: 11 total, 11 up, 11 in 2024-03-17T23:53:34.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:34 smithi191 ceph-mon[36708]: osdmap e75: 11 total, 11 up, 11 in 2024-03-17T23:53:35.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:35 smithi191 ceph-mon[36708]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-17T23:53:35.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:35 smithi191 ceph-mon[36708]: osdmap e76: 11 total, 11 up, 11 in 2024-03-17T23:53:35.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:35 smithi191 ceph-mon[36708]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:53:35.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:53:35.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:53:35.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:35 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:35 smithi012 ceph-mon[31541]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-17T23:53:35.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:35 smithi012 ceph-mon[31541]: osdmap e76: 11 total, 11 up, 11 in 2024-03-17T23:53:35.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:35 smithi012 ceph-mon[31541]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:53:35.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:53:35.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:53:35.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:35 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:35 smithi079 ceph-mon[36241]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-17T23:53:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:35 smithi079 ceph-mon[36241]: osdmap e76: 11 total, 11 up, 11 in 2024-03-17T23:53:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:35 smithi079 ceph-mon[36241]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:53:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-17T23:53:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-17T23:53:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:35 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:37 smithi012 ceph-mon[31541]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-03-17T23:53:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:37 smithi012 ceph-mon[31541]: from='client.? 172.21.15.191:0/3846090440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58"}]: dispatch 2024-03-17T23:53:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:37 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58"}]: dispatch 2024-03-17T23:53:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:37 smithi012 ceph-mon[31541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58"}]': finished 2024-03-17T23:53:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:37 smithi012 ceph-mon[31541]: osdmap e77: 12 total, 11 up, 12 in 2024-03-17T23:53:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:37.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:37 smithi079 ceph-mon[36241]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-03-17T23:53:37.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:37 smithi079 ceph-mon[36241]: from='client.? 172.21.15.191:0/3846090440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58"}]: dispatch 2024-03-17T23:53:37.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:37 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58"}]: dispatch 2024-03-17T23:53:37.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:37 smithi079 ceph-mon[36241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58"}]': finished 2024-03-17T23:53:37.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:37 smithi079 ceph-mon[36241]: osdmap e77: 12 total, 11 up, 12 in 2024-03-17T23:53:37.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:37 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:37.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:37 smithi191 ceph-mon[36708]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-03-17T23:53:37.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:37 smithi191 ceph-mon[36708]: from='client.? 172.21.15.191:0/3846090440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58"}]: dispatch 2024-03-17T23:53:37.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:37 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58"}]: dispatch 2024-03-17T23:53:37.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:37 smithi191 ceph-mon[36708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58"}]': finished 2024-03-17T23:53:37.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:37 smithi191 ceph-mon[36708]: osdmap e77: 12 total, 11 up, 12 in 2024-03-17T23:53:37.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:37 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:38.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:38 smithi012 ceph-mon[31541]: from='client.? 172.21.15.191:0/976490633' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:53:38.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:38 smithi079 ceph-mon[36241]: from='client.? 172.21.15.191:0/976490633' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:53:38.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:38 smithi191 ceph-mon[36708]: from='client.? 172.21.15.191:0/976490633' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-17T23:53:39.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:39 smithi012 ceph-mon[31541]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-03-17T23:53:39.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:39 smithi079 ceph-mon[36241]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-03-17T23:53:39.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:39 smithi191 ceph-mon[36708]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-03-17T23:53:41.494 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:41 smithi191 ceph-mon[36708]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-17T23:53:41.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:41 smithi012 ceph-mon[31541]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-17T23:53:41.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:41 smithi079 ceph-mon[36241]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-17T23:53:43.470 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:43 smithi191 ceph-mon[36708]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:43.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:43 smithi012 ceph-mon[31541]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:43.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:43 smithi079 ceph-mon[36241]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:45.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:45 smithi191 ceph-mon[36708]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:45.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:45 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-17T23:53:45.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:45 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:45.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:45 smithi191 ceph-mon[36708]: Deploying daemon osd.11 on smithi191 2024-03-17T23:53:45.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:45 smithi012 ceph-mon[31541]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:45.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:45 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-17T23:53:45.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:45 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:45.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:45 smithi012 ceph-mon[31541]: Deploying daemon osd.11 on smithi191 2024-03-17T23:53:45.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:45 smithi079 ceph-mon[36241]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:45.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:45 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-17T23:53:45.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:45 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:45.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:45 smithi079 ceph-mon[36241]: Deploying daemon osd.11 on smithi191 2024-03-17T23:53:47.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:47 smithi012 ceph-mon[31541]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:47.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:47 smithi079 ceph-mon[36241]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:47.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:47 smithi191 ceph-mon[36708]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:49.687 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:49 smithi191 ceph-mon[36708]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:49.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:49 smithi079 ceph-mon[36241]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:49.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:49 smithi012 ceph-mon[31541]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:52.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:51 smithi079 ceph-mon[36241]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:52.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:52.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:52.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:52.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:51 smithi191 ceph-mon[36708]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:52.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:52.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:52.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:52.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:51 smithi012 ceph-mon[31541]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:52.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:52.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:52.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:53 smithi191 ceph-mon[36708]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:54.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:53 smithi079 ceph-mon[36241]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:54.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:53 smithi012 ceph-mon[31541]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:54.277 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:54.276+0000 7f08b67fc700 1 -- 172.21.15.191:0/3013676826 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f08c0062710 con 0x7f08ac0a2890 2024-03-17T23:53:54.278 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 11 on host 'smithi191' 2024-03-17T23:53:54.279 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:54.278+0000 7f08c754f700 1 -- 172.21.15.191:0/3013676826 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f08ac0a2890 msgr2=0x7f08ac0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:54.279 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:54.278+0000 7f08c754f700 1 --2- 172.21.15.191:0/3013676826 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f08ac0a2890 0x7f08ac0a4d50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f08c0103360 tx=0x7f08bc009190 comp rx=0 tx=0).stop 2024-03-17T23:53:54.279 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:54.278+0000 7f08c754f700 1 -- 172.21.15.191:0/3013676826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08c0103ea0 msgr2=0x7f08c0074c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:54.280 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:54.278+0000 7f08c754f700 1 --2- 172.21.15.191:0/3013676826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08c0103ea0 0x7f08c0074c00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f08b800b070 tx=0x7f08b800be60 comp rx=0 tx=0).stop 2024-03-17T23:53:54.280 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:54.278+0000 7f08c754f700 1 -- 172.21.15.191:0/3013676826 shutdown_connections 2024-03-17T23:53:54.280 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:54.278+0000 7f08c754f700 1 --2- 172.21.15.191:0/3013676826 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f08ac0a2890 0x7f08ac0a4d50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:54.280 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:54.278+0000 7f08c754f700 1 --2- 172.21.15.191:0/3013676826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08c0103ea0 0x7f08c0074c00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:54.280 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:54.278+0000 7f08c754f700 1 --2- 172.21.15.191:0/3013676826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08c0102300 0x7f08c0076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:54.280 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:54.278+0000 7f08c754f700 1 --2- 172.21.15.191:0/3013676826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08c0103500 0x7f08c00746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:54.280 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:54.279+0000 7f08c754f700 1 -- 172.21.15.191:0/3013676826 >> 172.21.15.191:0/3013676826 conn(0x7f08c00fd8e0 msgr2=0x7f08c0106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:53:54.280 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:54.279+0000 7f08c754f700 1 -- 172.21.15.191:0/3013676826 shutdown_connections 2024-03-17T23:53:54.280 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:53:54.279+0000 7f08c754f700 1 -- 172.21.15.191:0/3013676826 wait complete. 2024-03-17T23:53:54.970 DEBUG:teuthology.orchestra.run.smithi191:osd.11> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.11.service 2024-03-17T23:53:54.973 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-17T23:53:54.973 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-17T23:53:55.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:55 smithi012 ceph-mon[31541]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:55.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:55.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:55.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:55 smithi012 ceph-mon[31541]: from='osd.11 [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-17T23:53:55.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:55 smithi012 ceph-mon[31541]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-17T23:53:55.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:55.468 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:53:55.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:55 smithi079 ceph-mon[36241]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:55.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:55.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:55.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:55 smithi079 ceph-mon[36241]: from='osd.11 [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-17T23:53:55.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:55 smithi079 ceph-mon[36241]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-17T23:53:55.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:55.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:55 smithi191 ceph-mon[36708]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:55.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:55.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:55.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:55 smithi191 ceph-mon[36708]: from='osd.11 [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-17T23:53:55.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:55 smithi191 ceph-mon[36708]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-17T23:53:55.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:56.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:56 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:56.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:56 smithi079 ceph-mon[36241]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-17T23:53:56.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:56 smithi079 ceph-mon[36241]: from='osd.11 [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:53:56.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:56 smithi079 ceph-mon[36241]: osdmap e78: 12 total, 11 up, 12 in 2024-03-17T23:53:56.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:56 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:56.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:56 smithi079 ceph-mon[36241]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:53:56.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:56 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:56.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:56 smithi191 ceph-mon[36708]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-17T23:53:56.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:56 smithi191 ceph-mon[36708]: from='osd.11 [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:53:56.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:56 smithi191 ceph-mon[36708]: osdmap e78: 12 total, 11 up, 12 in 2024-03-17T23:53:56.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:56 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:56.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:56 smithi191 ceph-mon[36708]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:53:56.592 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 17 23:53:56 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-17T23:53:56.241+0000 7fa0cd019700 -1 osd.11 0 waiting for initial osdmap 2024-03-17T23:53:56.592 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 17 23:53:56 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-17T23:53:56.256+0000 7fa0c8e0c700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:53:56.604 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:56 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:56.604 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:56 smithi012 ceph-mon[31541]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-17T23:53:56.604 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:56 smithi012 ceph-mon[31541]: from='osd.11 [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:53:56.604 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:56 smithi012 ceph-mon[31541]: osdmap e78: 12 total, 11 up, 12 in 2024-03-17T23:53:56.604 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:56 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:56.604 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:56 smithi012 ceph-mon[31541]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-17T23:53:57.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:57 smithi012 ceph-mon[31541]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:57.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:57 smithi012 ceph-mon[31541]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-17T23:53:57.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:57 smithi012 ceph-mon[31541]: osdmap e79: 12 total, 11 up, 12 in 2024-03-17T23:53:57.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:57.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:57.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:57 smithi191 ceph-mon[36708]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:57.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:57 smithi191 ceph-mon[36708]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-17T23:53:57.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:57 smithi191 ceph-mon[36708]: osdmap e79: 12 total, 11 up, 12 in 2024-03-17T23:53:57.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:57.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:57.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:57 smithi079 ceph-mon[36241]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:57.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:57 smithi079 ceph-mon[36241]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-17T23:53:57.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:57 smithi079 ceph-mon[36241]: osdmap e79: 12 total, 11 up, 12 in 2024-03-17T23:53:57.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:57.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:58.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.111+0000 7f2c19a00700 1 -- 172.21.15.12:0/2166087896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c14108950 msgr2=0x7f2c14108d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:58.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.111+0000 7f2c19a00700 1 --2- 172.21.15.12:0/2166087896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c14108950 0x7f2c14108d70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c040098f0 tx=0x7f2c04009c00 comp rx=0 tx=0).stop 2024-03-17T23:53:58.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.111+0000 7f2c19a00700 1 -- 172.21.15.12:0/2166087896 shutdown_connections 2024-03-17T23:53:58.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.111+0000 7f2c19a00700 1 --2- 172.21.15.12:0/2166087896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c1410a4f0 0x7f2c14110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:58.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.111+0000 7f2c19a00700 1 --2- 172.21.15.12:0/2166087896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c14108950 0x7f2c14108d70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:58.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.111+0000 7f2c19a00700 1 --2- 172.21.15.12:0/2166087896 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c14109b50 0x7f2c14109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:58.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.111+0000 7f2c19a00700 1 -- 172.21.15.12:0/2166087896 >> 172.21.15.12:0/2166087896 conn(0x7f2c14077920 msgr2=0x7f2c14079d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:53:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.111+0000 7f2c19a00700 1 -- 172.21.15.12:0/2166087896 shutdown_connections 2024-03-17T23:53:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.111+0000 7f2c19a00700 1 -- 172.21.15.12:0/2166087896 wait complete. 2024-03-17T23:53:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.112+0000 7f2c19a00700 1 Processor -- start 2024-03-17T23:53:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.112+0000 7f2c19a00700 1 -- start start 2024-03-17T23:53:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.113+0000 7f2c19a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c14108950 0x7f2c1419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:53:58.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.113+0000 7f2c19a00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c14109b50 0x7f2c1419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:53:58.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.113+0000 7f2c19a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c1410a4f0 0x7f2c141a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:53:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.113+0000 7f2c19a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2c14114560 con 0x7f2c14108950 2024-03-17T23:53:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.113+0000 7f2c19a00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2c141143e0 con 0x7f2c14109b50 2024-03-17T23:53:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.113+0000 7f2c19a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2c141146e0 con 0x7f2c1410a4f0 2024-03-17T23:53:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.113+0000 7f2c12a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c14108950 0x7f2c1419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:53:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.113+0000 7f2c13400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c1410a4f0 0x7f2c141a6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:53:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.113+0000 7f2c12a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c14108950 0x7f2c1419eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48822/0 (socket says 172.21.15.12:48822) 2024-03-17T23:53:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.113+0000 7f2c12a00700 1 -- 172.21.15.12:0/581118469 learned_addr learned my addr 172.21.15.12:0/581118469 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:53:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.113+0000 7f2c0a000700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c14109b50 0x7f2c1419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:53:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.114+0000 7f2c13400700 1 -- 172.21.15.12:0/581118469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c14109b50 msgr2=0x7f2c1419f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.114+0000 7f2c13400700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c14109b50 0x7f2c1419f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.114+0000 7f2c13400700 1 -- 172.21.15.12:0/581118469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c14108950 msgr2=0x7f2c1419eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.114+0000 7f2c13400700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c14108950 0x7f2c1419eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.114+0000 7f2c13400700 1 -- 172.21.15.12:0/581118469 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c141a6800 con 0x7f2c1410a4f0 2024-03-17T23:53:58.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.114+0000 7f2c12a00700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c14108950 0x7f2c1419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:53:58.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.114+0000 7f2c13400700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c1410a4f0 0x7f2c141a6160 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009f60 tx=0x7f2c0c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:53:58.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.114+0000 7f2c0be00700 1 -- 172.21.15.12:0/581118469 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c0092e0 con 0x7f2c1410a4f0 2024-03-17T23:53:58.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.114+0000 7f2c0be00700 1 -- 172.21.15.12:0/581118469 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c0c004bb0 con 0x7f2c1410a4f0 2024-03-17T23:53:58.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.114+0000 7f2c0be00700 1 -- 172.21.15.12:0/581118469 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c004020 con 0x7f2c1410a4f0 2024-03-17T23:53:58.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.114+0000 7f2c19a00700 1 -- 172.21.15.12:0/581118469 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c141a6a90 con 0x7f2c1410a4f0 2024-03-17T23:53:58.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.115+0000 7f2c19a00700 1 -- 172.21.15.12:0/581118469 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c141a6f80 con 0x7f2c1410a4f0 2024-03-17T23:53:58.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.116+0000 7f2c19a00700 1 -- 172.21.15.12:0/581118469 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c14068f90 con 0x7f2c1410a4f0 2024-03-17T23:53:58.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.117+0000 7f2c0be00700 1 -- 172.21.15.12:0/581118469 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2c0c01b030 con 0x7f2c1410a4f0 2024-03-17T23:53:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.119+0000 7f2c0be00700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2bf80a2910 0x7f2bf80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:53:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.119+0000 7f2c0be00700 1 -- 172.21.15.12:0/581118469 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7f2c0c0cdfe0 con 0x7f2c1410a4f0 2024-03-17T23:53:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.120+0000 7f2c12a00700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2bf80a2910 0x7f2bf80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:53:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.120+0000 7f2c12a00700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2bf80a2910 0x7f2bf80a4dd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400c010 tx=0x7f2c0401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:53:58.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.123+0000 7f2c0be00700 1 -- 172.21.15.12:0/581118469 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2c0c022020 con 0x7f2c1410a4f0 2024-03-17T23:53:58.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.358+0000 7f2c19a00700 1 -- 172.21.15.12:0/581118469 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f2c14062ee0 con 0x7f2c1410a4f0 2024-03-17T23:53:58.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.359+0000 7f2c0be00700 1 -- 172.21.15.12:0/581118469 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f2c0c010040 con 0x7f2c1410a4f0 2024-03-17T23:53:58.360 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:53:58.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.362+0000 7f2c19a00700 1 -- 172.21.15.12:0/581118469 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2bf80a2910 msgr2=0x7f2bf80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:58.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.362+0000 7f2c19a00700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2bf80a2910 0x7f2bf80a4dd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400c010 tx=0x7f2c0401a040 comp rx=0 tx=0).stop 2024-03-17T23:53:58.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.362+0000 7f2c19a00700 1 -- 172.21.15.12:0/581118469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c1410a4f0 msgr2=0x7f2c141a6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:53:58.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.362+0000 7f2c19a00700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c1410a4f0 0x7f2c141a6160 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009f60 tx=0x7f2c0c009f90 comp rx=0 tx=0).stop 2024-03-17T23:53:58.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.362+0000 7f2c19a00700 1 -- 172.21.15.12:0/581118469 shutdown_connections 2024-03-17T23:53:58.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.362+0000 7f2c19a00700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2bf80a2910 0x7f2bf80a4dd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:58.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.362+0000 7f2c19a00700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c1410a4f0 0x7f2c141a6160 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:58.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.363+0000 7f2c19a00700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c14108950 0x7f2c1419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:58.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.363+0000 7f2c19a00700 1 --2- 172.21.15.12:0/581118469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c14109b50 0x7f2c1419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:53:58.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.363+0000 7f2c19a00700 1 -- 172.21.15.12:0/581118469 >> 172.21.15.12:0/581118469 conn(0x7f2c14077920 msgr2=0x7f2c1410ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:53:58.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.363+0000 7f2c19a00700 1 -- 172.21.15.12:0/581118469 shutdown_connections 2024-03-17T23:53:58.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:53:58.363+0000 7f2c19a00700 1 -- 172.21.15.12:0/581118469 wait complete. 2024-03-17T23:53:58.468 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:58 smithi191 ceph-mon[36708]: purged_snaps scrub starts 2024-03-17T23:53:58.468 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:58 smithi191 ceph-mon[36708]: purged_snaps scrub ok 2024-03-17T23:53:58.468 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:58 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:58.468 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:58 smithi191 ceph-mon[36708]: osd.11 [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303] boot 2024-03-17T23:53:58.468 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:58 smithi191 ceph-mon[36708]: osdmap e80: 12 total, 12 up, 12 in 2024-03-17T23:53:58.468 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:58 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:58.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:58 smithi079 ceph-mon[36241]: purged_snaps scrub starts 2024-03-17T23:53:58.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:58 smithi079 ceph-mon[36241]: purged_snaps scrub ok 2024-03-17T23:53:58.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:58 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:58.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:58 smithi079 ceph-mon[36241]: osd.11 [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303] boot 2024-03-17T23:53:58.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:58 smithi079 ceph-mon[36241]: osdmap e80: 12 total, 12 up, 12 in 2024-03-17T23:53:58.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:58 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:58.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:58 smithi012 ceph-mon[31541]: purged_snaps scrub starts 2024-03-17T23:53:58.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:58 smithi012 ceph-mon[31541]: purged_snaps scrub ok 2024-03-17T23:53:58.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:58 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:58.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:58 smithi012 ceph-mon[31541]: osd.11 [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303] boot 2024-03-17T23:53:58.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:58 smithi012 ceph-mon[31541]: osdmap e80: 12 total, 12 up, 12 in 2024-03-17T23:53:58.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:58 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-17T23:53:59.225 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1710719637,"num_in_osds":12,"osd_in_since":1710719617,"num_remapped_pgs":0} 2024-03-17T23:53:59.225 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-17T23:53:59.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:59.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: osdmap e81: 12 total, 12 up, 12 in 2024-03-17T23:53:59.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/581118469' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-17T23:53:59.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: Detected new or changed devices on smithi191 2024-03-17T23:53:59.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:59.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:59.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:59.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:59.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:59.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:59.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi191 to 4270M 2024-03-17T23:53:59.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:59.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:59.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:59.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:59.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:53:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:59.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:59.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: osdmap e81: 12 total, 12 up, 12 in 2024-03-17T23:53:59.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/581118469' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-17T23:53:59.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: Detected new or changed devices on smithi191 2024-03-17T23:53:59.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:59.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:59.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:59.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:59.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:59.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:59.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi191 to 4270M 2024-03-17T23:53:59.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:59.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:59.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:59.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:59.547 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:53:59 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:59.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-17T23:53:59.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: osdmap e81: 12 total, 12 up, 12 in 2024-03-17T23:53:59.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/581118469' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-17T23:53:59.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: Detected new or changed devices on smithi191 2024-03-17T23:53:59.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:59.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-17T23:53:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi191 to 4270M 2024-03-17T23:53:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:53:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:53:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:53:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:53:59 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:53:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:54:00.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:00 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:00.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:00 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:54:00.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:00 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:00.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:00 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:00.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:00 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:54:00.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:00 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:00.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:00 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:00.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:00 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:54:00.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:00 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.904+0000 7f6467400700 1 -- 172.21.15.12:0/1389308450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6460108300 msgr2=0x7f64601086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.904+0000 7f6467400700 1 --2- 172.21.15.12:0/1389308450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6460108300 0x7f64601086e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f64540099c0 tx=0x7f6454009cd0 comp rx=0 tx=0).stop 2024-03-17T23:54:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.904+0000 7f6467400700 1 -- 172.21.15.12:0/1389308450 shutdown_connections 2024-03-17T23:54:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.904+0000 7f6467400700 1 --2- 172.21.15.12:0/1389308450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6460102ce0 0x7f646010f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.904+0000 7f6467400700 1 --2- 172.21.15.12:0/1389308450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6460108300 0x7f64601086e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.904+0000 7f6467400700 1 --2- 172.21.15.12:0/1389308450 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6460102320 0x7f64601027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.904+0000 7f6467400700 1 -- 172.21.15.12:0/1389308450 >> 172.21.15.12:0/1389308450 conn(0x7f6460079400 msgr2=0x7f6460079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.904+0000 7f6467400700 1 -- 172.21.15.12:0/1389308450 shutdown_connections 2024-03-17T23:54:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.904+0000 7f6467400700 1 -- 172.21.15.12:0/1389308450 wait complete. 2024-03-17T23:54:01.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.905+0000 7f6467400700 1 Processor -- start 2024-03-17T23:54:01.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.906+0000 7f6467400700 1 -- start start 2024-03-17T23:54:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.906+0000 7f6467400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6460102320 0x7f64601a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.906+0000 7f6467400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6460102ce0 0x7f64601a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.906+0000 7f6467400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6460108300 0x7f646019aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.906+0000 7f6467400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f64601125d0 con 0x7f6460108300 2024-03-17T23:54:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.906+0000 7f6467400700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6460112450 con 0x7f6460102ce0 2024-03-17T23:54:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.906+0000 7f6467400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6460112750 con 0x7f6460102320 2024-03-17T23:54:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.906+0000 7f6465400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6460108300 0x7f646019aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.906+0000 7f6465400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6460108300 0x7f646019aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48848/0 (socket says 172.21.15.12:48848) 2024-03-17T23:54:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.907+0000 7f6464a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6460102320 0x7f64601a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.907+0000 7f6465400700 1 -- 172.21.15.12:0/1338482340 learned_addr learned my addr 172.21.15.12:0/1338482340 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.906+0000 7f645fe00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6460102ce0 0x7f64601a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.907+0000 7f6464a00700 1 -- 172.21.15.12:0/1338482340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6460102ce0 msgr2=0x7f64601a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.907+0000 7f6464a00700 1 --2- 172.21.15.12:0/1338482340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6460102ce0 0x7f64601a0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.907+0000 7f6464a00700 1 -- 172.21.15.12:0/1338482340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6460108300 msgr2=0x7f646019aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.907+0000 7f6464a00700 1 --2- 172.21.15.12:0/1338482340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6460108300 0x7f646019aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.907+0000 7f6464a00700 1 -- 172.21.15.12:0/1338482340 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f646019b220 con 0x7f6460102320 2024-03-17T23:54:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.907+0000 7f645fe00700 1 --2- 172.21.15.12:0/1338482340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6460102ce0 0x7f64601a0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.907+0000 7f6464a00700 1 --2- 172.21.15.12:0/1338482340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6460102320 0x7f64601a0900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f64540098e0 tx=0x7f6454003a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.908+0000 7f645d600700 1 -- 172.21.15.12:0/1338482340 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6454003680 con 0x7f6460102320 2024-03-17T23:54:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.908+0000 7f645d600700 1 -- 172.21.15.12:0/1338482340 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64540055a0 con 0x7f6460102320 2024-03-17T23:54:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.908+0000 7f645d600700 1 -- 172.21.15.12:0/1338482340 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645400fc90 con 0x7f6460102320 2024-03-17T23:54:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.908+0000 7f6467400700 1 -- 172.21.15.12:0/1338482340 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f646019b4b0 con 0x7f6460102320 2024-03-17T23:54:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.908+0000 7f6467400700 1 -- 172.21.15.12:0/1338482340 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64601a77d0 con 0x7f6460102320 2024-03-17T23:54:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.909+0000 7f6467400700 1 -- 172.21.15.12:0/1338482340 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6460068f90 con 0x7f6460102320 2024-03-17T23:54:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.911+0000 7f645d600700 1 -- 172.21.15.12:0/1338482340 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f645400f460 con 0x7f6460102320 2024-03-17T23:54:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.912+0000 7f645d600700 1 --2- 172.21.15.12:0/1338482340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f64480a2ab0 0x7f64480a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.912+0000 7f645d600700 1 -- 172.21.15.12:0/1338482340 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f64540cf370 con 0x7f6460102320 2024-03-17T23:54:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.912+0000 7f645fe00700 1 --2- 172.21.15.12:0/1338482340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f64480a2ab0 0x7f64480a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.913+0000 7f645fe00700 1 --2- 172.21.15.12:0/1338482340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f64480a2ab0 0x7f64480a4f70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6460076640 tx=0x7f644c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:01.916+0000 7f645d600700 1 -- 172.21.15.12:0/1338482340 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6454013070 con 0x7f6460102320 2024-03-17T23:54:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:01 smithi191 ceph-mon[36708]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:02.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.142+0000 7f6467400700 1 -- 172.21.15.12:0/1338482340 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f646019bff0 con 0x7f6460102320 2024-03-17T23:54:02.143 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:01 smithi012 ceph-mon[31541]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:02.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.144+0000 7f645d600700 1 -- 172.21.15.12:0/1338482340 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18372 (secure 0 0 0) 0x7f646019bff0 con 0x7f6460102320 2024-03-17T23:54:02.145 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:02.146 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:53:58.251750+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:54:02.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.148+0000 7f6467400700 1 -- 172.21.15.12:0/1338482340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f64480a2ab0 msgr2=0x7f64480a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:02.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.148+0000 7f6467400700 1 --2- 172.21.15.12:0/1338482340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f64480a2ab0 0x7f64480a4f70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6460076640 tx=0x7f644c009260 comp rx=0 tx=0).stop 2024-03-17T23:54:02.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.148+0000 7f6467400700 1 -- 172.21.15.12:0/1338482340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6460102320 msgr2=0x7f64601a0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:02.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.148+0000 7f6467400700 1 --2- 172.21.15.12:0/1338482340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6460102320 0x7f64601a0900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f64540098e0 tx=0x7f6454003a90 comp rx=0 tx=0).stop 2024-03-17T23:54:02.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.148+0000 7f6467400700 1 -- 172.21.15.12:0/1338482340 shutdown_connections 2024-03-17T23:54:02.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.148+0000 7f6467400700 1 --2- 172.21.15.12:0/1338482340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f64480a2ab0 0x7f64480a4f70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:02.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.148+0000 7f6467400700 1 --2- 172.21.15.12:0/1338482340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6460108300 0x7f646019aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:02.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.148+0000 7f6467400700 1 --2- 172.21.15.12:0/1338482340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6460102320 0x7f64601a0900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:02.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.148+0000 7f6467400700 1 --2- 172.21.15.12:0/1338482340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6460102ce0 0x7f64601a0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:02.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.148+0000 7f6467400700 1 -- 172.21.15.12:0/1338482340 >> 172.21.15.12:0/1338482340 conn(0x7f6460079400 msgr2=0x7f646010d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:02.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.148+0000 7f6467400700 1 -- 172.21.15.12:0/1338482340 shutdown_connections 2024-03-17T23:54:02.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:02.148+0000 7f6467400700 1 -- 172.21.15.12:0/1338482340 wait complete. 2024-03-17T23:54:02.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:01 smithi079 ceph-mon[36241]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:03.006 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:49:38.640282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:54:03.006 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-17T23:54:03.099 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:02 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1338482340' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:54:03.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:02 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1338482340' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:54:03.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:02 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1338482340' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:54:03.484 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:54:03.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:03 smithi012 ceph-mon[31541]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:04.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:03 smithi079 ceph-mon[36241]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:03 smithi191 ceph-mon[36708]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:05.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.755+0000 7f4406a00700 1 -- 172.21.15.12:0/4254550054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4400074e20 msgr2=0x7f440010e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:05.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.755+0000 7f4406a00700 1 --2- 172.21.15.12:0/4254550054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4400074e20 0x7f440010e790 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f43ec009990 tx=0x7f43ec009ca0 comp rx=0 tx=0).stop 2024-03-17T23:54:05.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.756+0000 7f4406a00700 1 -- 172.21.15.12:0/4254550054 shutdown_connections 2024-03-17T23:54:05.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.756+0000 7f4406a00700 1 --2- 172.21.15.12:0/4254550054 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f440010ecd0 0x7f44001110c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:05.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.756+0000 7f4406a00700 1 --2- 172.21.15.12:0/4254550054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4400074500 0x7f44000748e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:05.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.756+0000 7f4406a00700 1 --2- 172.21.15.12:0/4254550054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4400074e20 0x7f440010e790 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:05.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.756+0000 7f4406a00700 1 -- 172.21.15.12:0/4254550054 >> 172.21.15.12:0/4254550054 conn(0x7f4400079400 msgr2=0x7f4400079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:05.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.756+0000 7f4406a00700 1 -- 172.21.15.12:0/4254550054 shutdown_connections 2024-03-17T23:54:05.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.756+0000 7f4406a00700 1 -- 172.21.15.12:0/4254550054 wait complete. 2024-03-17T23:54:05.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.757+0000 7f4406a00700 1 Processor -- start 2024-03-17T23:54:05.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.757+0000 7f4406a00700 1 -- start start 2024-03-17T23:54:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.757+0000 7f4406a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4400074500 0x7f4400107dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.757+0000 7f4406a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4400074e20 0x7f4400108310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.757+0000 7f4406a00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f440010ecd0 0x7f4400102f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f4406a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4400114820 con 0x7f4400074e20 2024-03-17T23:54:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f4406a00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f44001146a0 con 0x7f440010ecd0 2024-03-17T23:54:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f4406a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f44001149a0 con 0x7f4400074500 2024-03-17T23:54:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f43ff400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4400074e20 0x7f4400108310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f43ff400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4400074e20 0x7f4400108310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46142/0 (socket says 172.21.15.12:46142) 2024-03-17T23:54:05.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f43ff400700 1 -- 172.21.15.12:0/77110721 learned_addr learned my addr 172.21.15.12:0/77110721 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f43ffe00700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4400074500 0x7f4400107dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f4404a00700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f440010ecd0 0x7f4400102f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f43ff400700 1 -- 172.21.15.12:0/77110721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4400074500 msgr2=0x7f4400107dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f43ff400700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4400074500 0x7f4400107dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f43ff400700 1 -- 172.21.15.12:0/77110721 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f440010ecd0 msgr2=0x7f4400102f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f43ff400700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f440010ecd0 0x7f4400102f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f43ff400700 1 -- 172.21.15.12:0/77110721 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4400103840 con 0x7f4400074e20 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.758+0000 7f43ffe00700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4400074500 0x7f4400107dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.759+0000 7f43ff400700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4400074e20 0x7f4400108310 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f43ec000c00 tx=0x7f43ec00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.759+0000 7f43fcc00700 1 -- 172.21.15.12:0/77110721 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec027070 con 0x7f4400074e20 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.759+0000 7f43fcc00700 1 -- 172.21.15.12:0/77110721 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43ec003d70 con 0x7f4400074e20 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.759+0000 7f4406a00700 1 -- 172.21.15.12:0/77110721 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4400103ad0 con 0x7f4400074e20 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.759+0000 7f43fcc00700 1 -- 172.21.15.12:0/77110721 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec00f460 con 0x7f4400074e20 2024-03-17T23:54:05.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.759+0000 7f4406a00700 1 -- 172.21.15.12:0/77110721 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f440010dcc0 con 0x7f4400074e20 2024-03-17T23:54:05.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.760+0000 7f4406a00700 1 -- 172.21.15.12:0/77110721 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4400075bb0 con 0x7f4400074e20 2024-03-17T23:54:05.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.762+0000 7f43fcc00700 1 -- 172.21.15.12:0/77110721 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f43ec02a070 con 0x7f4400074e20 2024-03-17T23:54:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.764+0000 7f43fcc00700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f43e40a2910 0x7f43e40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.764+0000 7f43fcc00700 1 -- 172.21.15.12:0/77110721 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f43ec0cf450 con 0x7f4400074e20 2024-03-17T23:54:05.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.764+0000 7f43ffe00700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f43e40a2910 0x7f43e40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:05.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.765+0000 7f43ffe00700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f43e40a2910 0x7f43e40a4dd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f43f0011490 tx=0x7f43f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:05.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.769+0000 7f43fcc00700 1 -- 172.21.15.12:0/77110721 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f43ec05e740 con 0x7f4400074e20 2024-03-17T23:54:05.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.995+0000 7f4406a00700 1 -- 172.21.15.12:0/77110721 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f440010dfe0 con 0x7f4400074e20 2024-03-17T23:54:05.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.996+0000 7f43fcc00700 1 -- 172.21.15.12:0/77110721 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f43ec013070 con 0x7f4400074e20 2024-03-17T23:54:05.997 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-17T23:54:06.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.999+0000 7f4406a00700 1 -- 172.21.15.12:0/77110721 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f43e40a2910 msgr2=0x7f43e40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:06.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.999+0000 7f4406a00700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f43e40a2910 0x7f43e40a4dd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f43f0011490 tx=0x7f43f0009260 comp rx=0 tx=0).stop 2024-03-17T23:54:06.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.999+0000 7f4406a00700 1 -- 172.21.15.12:0/77110721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4400074e20 msgr2=0x7f4400108310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:06.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.999+0000 7f4406a00700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4400074e20 0x7f4400108310 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f43ec000c00 tx=0x7f43ec00bab0 comp rx=0 tx=0).stop 2024-03-17T23:54:06.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.999+0000 7f4406a00700 1 -- 172.21.15.12:0/77110721 shutdown_connections 2024-03-17T23:54:06.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.999+0000 7f4406a00700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f43e40a2910 0x7f43e40a4dd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:06.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.999+0000 7f4406a00700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f440010ecd0 0x7f4400102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:06.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.999+0000 7f4406a00700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4400074500 0x7f4400107dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:06.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.999+0000 7f4406a00700 1 --2- 172.21.15.12:0/77110721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4400074e20 0x7f4400108310 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:06.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.999+0000 7f4406a00700 1 -- 172.21.15.12:0/77110721 >> 172.21.15.12:0/77110721 conn(0x7f4400079400 msgr2=0x7f44000ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.999+0000 7f4406a00700 1 -- 172.21.15.12:0/77110721 shutdown_connections 2024-03-17T23:54:06.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:05.999+0000 7f4406a00700 1 -- 172.21.15.12:0/77110721 wait complete. 2024-03-17T23:54:06.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:05 smithi012 ceph-mon[31541]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:06.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:05 smithi079 ceph-mon[36241]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:06.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:05 smithi191 ceph-mon[36708]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:06.881 INFO:tasks.cephadm:Adding mds.a on smithi012 2024-03-17T23:54:06.882 INFO:tasks.cephadm:Adding mds.d on smithi012 2024-03-17T23:54:06.882 INFO:tasks.cephadm:Adding mds.g on smithi012 2024-03-17T23:54:06.882 INFO:tasks.cephadm:Adding mds.j on smithi012 2024-03-17T23:54:06.882 INFO:tasks.cephadm:Adding mds.b on smithi079 2024-03-17T23:54:06.882 INFO:tasks.cephadm:Adding mds.e on smithi079 2024-03-17T23:54:06.882 INFO:tasks.cephadm:Adding mds.h on smithi079 2024-03-17T23:54:06.882 INFO:tasks.cephadm:Adding mds.k on smithi079 2024-03-17T23:54:06.882 INFO:tasks.cephadm:Adding mds.c on smithi191 2024-03-17T23:54:06.882 INFO:tasks.cephadm:Adding mds.f on smithi191 2024-03-17T23:54:06.882 INFO:tasks.cephadm:Adding mds.i on smithi191 2024-03-17T23:54:06.882 INFO:tasks.cephadm:Adding mds.l on smithi191 2024-03-17T23:54:06.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l' 2024-03-17T23:54:07.130 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-17T23:54:07.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:06 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/77110721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:54:07.242 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:06 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/77110721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:54:07.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:06 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/77110721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:54:08.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:07 smithi012 ceph-mon[31541]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:08.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:07 smithi191 ceph-mon[36708]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:08.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:07 smithi079 ceph-mon[36241]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:08.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.833+0000 7f8fd476e700 1 -- 172.21.15.191:0/3281019391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fcc102da0 msgr2=0x7f8fcc109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:08.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.833+0000 7f8fd476e700 1 --2- 172.21.15.191:0/3281019391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fcc102da0 0x7f8fcc109650 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009990 tx=0x7f8fc4009ca0 comp rx=0 tx=0).stop 2024-03-17T23:54:08.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.833+0000 7f8fd476e700 1 -- 172.21.15.191:0/3281019391 shutdown_connections 2024-03-17T23:54:08.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.833+0000 7f8fd476e700 1 --2- 172.21.15.191:0/3281019391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fcc102da0 0x7f8fcc109650 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:08.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.833+0000 7f8fd476e700 1 --2- 172.21.15.191:0/3281019391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fcc101200 0x7f8fcc101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:08.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.833+0000 7f8fd476e700 1 --2- 172.21.15.191:0/3281019391 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8fcc102400 0x7f8fcc102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:08.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.833+0000 7f8fd476e700 1 -- 172.21.15.191:0/3281019391 >> 172.21.15.191:0/3281019391 conn(0x7f8fcc0fc7c0 msgr2=0x7f8fcc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.833+0000 7f8fd476e700 1 -- 172.21.15.191:0/3281019391 shutdown_connections 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.833+0000 7f8fd476e700 1 -- 172.21.15.191:0/3281019391 wait complete. 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd476e700 1 Processor -- start 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd476e700 1 -- start start 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd476e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fcc101200 0x7f8fcc198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd476e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fcc102400 0x7f8fcc198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd476e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8fcc102da0 0x7f8fcc19fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd476e700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc10ce90 con 0x7f8fcc102400 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd476e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc10cd10 con 0x7f8fcc102da0 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd476e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc10d010 con 0x7f8fcc101200 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd250a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fcc101200 0x7f8fcc198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd250a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fcc101200 0x7f8fcc198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:53226/0 (socket says 172.21.15.191:53226) 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd250a700 1 -- 172.21.15.191:0/2101072796 learned_addr learned my addr 172.21.15.191:0/2101072796 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-17T23:54:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd2d0b700 1 --2- 172.21.15.191:0/2101072796 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8fcc102da0 0x7f8fcc19fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:08.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd250a700 1 -- 172.21.15.191:0/2101072796 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8fcc102da0 msgr2=0x7f8fcc19fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:08.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd250a700 1 --2- 172.21.15.191:0/2101072796 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8fcc102da0 0x7f8fcc19fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:08.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd250a700 1 -- 172.21.15.191:0/2101072796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fcc102400 msgr2=0x7f8fcc198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:08.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd250a700 1 --2- 172.21.15.191:0/2101072796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fcc102400 0x7f8fcc198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:08.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd250a700 1 -- 172.21.15.191:0/2101072796 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fcc1a0170 con 0x7f8fcc101200 2024-03-17T23:54:08.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.834+0000 7f8fd250a700 1 --2- 172.21.15.191:0/2101072796 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fcc101200 0x7f8fcc198510 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00b9f0 tx=0x7f8fbc00bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:08.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.835+0000 7f8fcb7fe700 1 -- 172.21.15.191:0/2101072796 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc019070 con 0x7f8fcc101200 2024-03-17T23:54:08.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.835+0000 7f8fd476e700 1 -- 172.21.15.191:0/2101072796 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fcc1a0460 con 0x7f8fcc101200 2024-03-17T23:54:08.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.835+0000 7f8fcb7fe700 1 -- 172.21.15.191:0/2101072796 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fbc00ce30 con 0x7f8fcc101200 2024-03-17T23:54:08.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.835+0000 7f8fcb7fe700 1 -- 172.21.15.191:0/2101072796 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc0125a0 con 0x7f8fcc101200 2024-03-17T23:54:08.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.835+0000 7f8fd476e700 1 -- 172.21.15.191:0/2101072796 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fcc1a0a70 con 0x7f8fcc101200 2024-03-17T23:54:08.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.836+0000 7f8fd476e700 1 -- 172.21.15.191:0/2101072796 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fcc068f90 con 0x7f8fcc101200 2024-03-17T23:54:08.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.837+0000 7f8fcb7fe700 1 -- 172.21.15.191:0/2101072796 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8fbc01c070 con 0x7f8fcc101200 2024-03-17T23:54:08.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.838+0000 7f8fcb7fe700 1 --2- 172.21.15.191:0/2101072796 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8fb40a2890 0x7f8fb40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:08.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.838+0000 7f8fcb7fe700 1 -- 172.21.15.191:0/2101072796 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8fbc0ce3f0 con 0x7f8fcc101200 2024-03-17T23:54:08.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.838+0000 7f8fd1d09700 1 --2- 172.21.15.191:0/2101072796 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8fb40a2890 0x7f8fb40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:08.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.839+0000 7f8fd1d09700 1 --2- 172.21.15.191:0/2101072796 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8fb40a2890 0x7f8fb40a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0005950 tx=0x7f8fc00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:08.842 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:08.842+0000 7f8fcb7fe700 1 -- 172.21.15.191:0/2101072796 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8fbc023070 con 0x7f8fcc101200 2024-03-17T23:54:09.048 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.047+0000 7f8fd476e700 1 -- 172.21.15.191:0/2101072796 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}) -- 0x7f8fcc0fe8b0 con 0x7f8fb40a2890 2024-03-17T23:54:09.063 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.063+0000 7f8fcb7fe700 1 -- 172.21.15.191:0/2101072796 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f8fcc0fe8b0 con 0x7f8fb40a2890 2024-03-17T23:54:09.064 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mds.all update... 2024-03-17T23:54:09.066 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.066+0000 7f8fd476e700 1 -- 172.21.15.191:0/2101072796 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8fb40a2890 msgr2=0x7f8fb40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:09.066 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.066+0000 7f8fd476e700 1 --2- 172.21.15.191:0/2101072796 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8fb40a2890 0x7f8fb40a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0005950 tx=0x7f8fc00058e0 comp rx=0 tx=0).stop 2024-03-17T23:54:09.066 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.066+0000 7f8fd476e700 1 -- 172.21.15.191:0/2101072796 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fcc101200 msgr2=0x7f8fcc198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:09.066 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.066+0000 7f8fd476e700 1 --2- 172.21.15.191:0/2101072796 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fcc101200 0x7f8fcc198510 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00b9f0 tx=0x7f8fbc00bd00 comp rx=0 tx=0).stop 2024-03-17T23:54:09.066 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.066+0000 7f8fd476e700 1 -- 172.21.15.191:0/2101072796 shutdown_connections 2024-03-17T23:54:09.066 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.066+0000 7f8fd476e700 1 --2- 172.21.15.191:0/2101072796 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8fb40a2890 0x7f8fb40a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:09.066 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.066+0000 7f8fd476e700 1 --2- 172.21.15.191:0/2101072796 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8fcc102da0 0x7f8fcc19fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:09.066 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.066+0000 7f8fd476e700 1 --2- 172.21.15.191:0/2101072796 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fcc101200 0x7f8fcc198510 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:09.067 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.066+0000 7f8fd476e700 1 --2- 172.21.15.191:0/2101072796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8fcc102400 0x7f8fcc198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:09.067 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.066+0000 7f8fd476e700 1 -- 172.21.15.191:0/2101072796 >> 172.21.15.191:0/2101072796 conn(0x7f8fcc0fc7c0 msgr2=0x7f8fcc0fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:09.067 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.066+0000 7f8fd476e700 1 -- 172.21.15.191:0/2101072796 shutdown_connections 2024-03-17T23:54:09.067 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-17T23:54:09.066+0000 7f8fd476e700 1 -- 172.21.15.191:0/2101072796 wait complete. 2024-03-17T23:54:09.782 DEBUG:teuthology.orchestra.run.smithi012:mds.a> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.a.service 2024-03-17T23:54:09.788 DEBUG:teuthology.orchestra.run.smithi012:mds.d> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.d.service 2024-03-17T23:54:09.833 DEBUG:teuthology.orchestra.run.smithi012:mds.g> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.g.service 2024-03-17T23:54:09.878 DEBUG:teuthology.orchestra.run.smithi012:mds.j> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.j.service 2024-03-17T23:54:09.923 DEBUG:teuthology.orchestra.run.smithi079:mds.b> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.b.service 2024-03-17T23:54:09.926 DEBUG:teuthology.orchestra.run.smithi079:mds.e> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.e.service 2024-03-17T23:54:09.968 DEBUG:teuthology.orchestra.run.smithi079:mds.h> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.h.service 2024-03-17T23:54:10.011 DEBUG:teuthology.orchestra.run.smithi079:mds.k> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.k.service 2024-03-17T23:54:10.055 DEBUG:teuthology.orchestra.run.smithi191:mds.c> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.c.service 2024-03-17T23:54:10.058 DEBUG:teuthology.orchestra.run.smithi191:mds.f> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.f.service 2024-03-17T23:54:10.083 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:09 smithi191 ceph-mon[36708]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:10.083 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:09 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:10.083 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:09 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:54:10.084 DEBUG:teuthology.orchestra.run.smithi191:mds.i> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.i.service 2024-03-17T23:54:10.126 DEBUG:teuthology.orchestra.run.smithi191:mds.l> sudo journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.l.service 2024-03-17T23:54:10.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:09 smithi012 ceph-mon[31541]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:10.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:09 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:54:10.169 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-17T23:54:10.170 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:54:10.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:09 smithi079 ceph-mon[36241]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:10.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:09 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:10.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:09 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:54:10.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.386+0000 7eff32a00640 1 -- 172.21.15.12:0/4247698620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff2c103a00 msgr2=0x7eff2c103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:10.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.386+0000 7eff32a00640 1 --2- 172.21.15.12:0/4247698620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff2c103a00 0x7eff2c103e20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7eff1c009900 tx=0x7eff1c02bc50 comp rx=0 tx=0).stop 2024-03-17T23:54:10.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.386+0000 7eff32a00640 1 -- 172.21.15.12:0/4247698620 shutdown_connections 2024-03-17T23:54:10.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.386+0000 7eff32a00640 1 --2- 172.21.15.12:0/4247698620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff2c1055b0 0x7eff2c10be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.386+0000 7eff32a00640 1 --2- 172.21.15.12:0/4247698620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff2c104bf0 0x7eff2c105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.386+0000 7eff32a00640 1 --2- 172.21.15.12:0/4247698620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff2c103a00 0x7eff2c103e20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.386+0000 7eff32a00640 1 -- 172.21.15.12:0/4247698620 >> 172.21.15.12:0/4247698620 conn(0x7eff2c0fef90 msgr2=0x7eff2c1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:10.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.387+0000 7eff32a00640 1 -- 172.21.15.12:0/4247698620 shutdown_connections 2024-03-17T23:54:10.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.387+0000 7eff32a00640 1 -- 172.21.15.12:0/4247698620 wait complete. 2024-03-17T23:54:10.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.388+0000 7eff32a00640 1 Processor -- start 2024-03-17T23:54:10.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.388+0000 7eff32a00640 1 -- start start 2024-03-17T23:54:10.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.388+0000 7eff32a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff2c103a00 0x7eff2c199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:10.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.388+0000 7eff32a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff2c104bf0 0x7eff2c19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:10.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.388+0000 7eff32a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff2c1055b0 0x7eff2c1a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:10.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.388+0000 7eff32a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7eff2c10f680 con 0x7eff2c1055b0 2024-03-17T23:54:10.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.388+0000 7eff32a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7eff2c10f500 con 0x7eff2c103a00 2024-03-17T23:54:10.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.388+0000 7eff32a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7eff2c10f800 con 0x7eff2c104bf0 2024-03-17T23:54:10.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.388+0000 7eff30a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff2c1055b0 0x7eff2c1a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:10.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.388+0000 7eff23e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff2c104bf0 0x7eff2c19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:10.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.388+0000 7eff23e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff2c104bf0 0x7eff2c19a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:41000/0 (socket says 172.21.15.12:41000) 2024-03-17T23:54:10.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.388+0000 7eff23e00640 1 -- 172.21.15.12:0/2711869965 learned_addr learned my addr 172.21.15.12:0/2711869965 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:10.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.388+0000 7eff2be00640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff2c103a00 0x7eff2c199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:10.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.389+0000 7eff30a00640 1 -- 172.21.15.12:0/2711869965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff2c104bf0 msgr2=0x7eff2c19a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:10.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.389+0000 7eff30a00640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff2c104bf0 0x7eff2c19a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.389+0000 7eff30a00640 1 -- 172.21.15.12:0/2711869965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff2c103a00 msgr2=0x7eff2c199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:10.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.389+0000 7eff30a00640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff2c103a00 0x7eff2c199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.389+0000 7eff30a00640 1 -- 172.21.15.12:0/2711869965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff2c1a1990 con 0x7eff2c1055b0 2024-03-17T23:54:10.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.389+0000 7eff2be00640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff2c103a00 0x7eff2c199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:10.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.389+0000 7eff30a00640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff2c1055b0 0x7eff2c1a1320 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7eff24009fd0 tx=0x7eff2400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:10.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.389+0000 7eff29600640 1 -- 172.21.15.12:0/2711869965 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2400ce20 con 0x7eff2c1055b0 2024-03-17T23:54:10.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.389+0000 7eff29600640 1 -- 172.21.15.12:0/2711869965 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff24005020 con 0x7eff2c1055b0 2024-03-17T23:54:10.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.389+0000 7eff32a00640 1 -- 172.21.15.12:0/2711869965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff2c1a1c80 con 0x7eff2c1055b0 2024-03-17T23:54:10.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.389+0000 7eff29600640 1 -- 172.21.15.12:0/2711869965 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff24010720 con 0x7eff2c1055b0 2024-03-17T23:54:10.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.390+0000 7eff32a00640 1 -- 172.21.15.12:0/2711869965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff2c1a21d0 con 0x7eff2c1055b0 2024-03-17T23:54:10.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.391+0000 7eff32a00640 1 -- 172.21.15.12:0/2711869965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff2c10ad20 con 0x7eff2c1055b0 2024-03-17T23:54:10.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.393+0000 7eff29600640 1 -- 172.21.15.12:0/2711869965 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7eff24005540 con 0x7eff2c1055b0 2024-03-17T23:54:10.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.394+0000 7eff29600640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7eff100a2800 0x7eff100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:10.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.394+0000 7eff29600640 1 -- 172.21.15.12:0/2711869965 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7eff240cec90 con 0x7eff2c1055b0 2024-03-17T23:54:10.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.394+0000 7eff2be00640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7eff100a2800 0x7eff100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:10.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.395+0000 7eff2be00640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7eff100a2800 0x7eff100a4cc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7eff1c000c00 tx=0x7eff1c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:10.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.399+0000 7eff29600640 1 -- 172.21.15.12:0/2711869965 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7eff2409c130 con 0x7eff2c1055b0 2024-03-17T23:54:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.601+0000 7eff32a00640 1 -- 172.21.15.12:0/2711869965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7eff2c062480 con 0x7eff2c1055b0 2024-03-17T23:54:10.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.602+0000 7eff29600640 1 -- 172.21.15.12:0/2711869965 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18372 (secure 0 0 0) 0x7eff2409ba30 con 0x7eff2c1055b0 2024-03-17T23:54:10.603 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:10.603 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:53:58.251750+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:54:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.605+0000 7eff32a00640 1 -- 172.21.15.12:0/2711869965 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7eff100a2800 msgr2=0x7eff100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.605+0000 7eff32a00640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7eff100a2800 0x7eff100a4cc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7eff1c000c00 tx=0x7eff1c005fd0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.605+0000 7eff32a00640 1 -- 172.21.15.12:0/2711869965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff2c1055b0 msgr2=0x7eff2c1a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.605+0000 7eff32a00640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff2c1055b0 0x7eff2c1a1320 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7eff24009fd0 tx=0x7eff2400ef10 comp rx=0 tx=0).stop 2024-03-17T23:54:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.605+0000 7eff32a00640 1 -- 172.21.15.12:0/2711869965 shutdown_connections 2024-03-17T23:54:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.605+0000 7eff32a00640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7eff100a2800 0x7eff100a4cc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.605+0000 7eff32a00640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff2c1055b0 0x7eff2c1a1320 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.605+0000 7eff32a00640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff2c104bf0 0x7eff2c19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.605+0000 7eff32a00640 1 --2- 172.21.15.12:0/2711869965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff2c103a00 0x7eff2c199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.605+0000 7eff32a00640 1 -- 172.21.15.12:0/2711869965 >> 172.21.15.12:0/2711869965 conn(0x7eff2c0fef90 msgr2=0x7eff2c107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.605+0000 7eff32a00640 1 -- 172.21.15.12:0/2711869965 shutdown_connections 2024-03-17T23:54:10.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.605+0000 7eff32a00640 1 -- 172.21.15.12:0/2711869965 wait complete. 2024-03-17T23:54:10.627 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:49:38.640282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:54:10.627 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:54:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.860+0000 7ff9a2800640 1 -- 172.21.15.12:0/3668368912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff99c103a00 msgr2=0x7ff99c103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.860+0000 7ff9a2800640 1 --2- 172.21.15.12:0/3668368912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff99c103a00 0x7ff99c103e20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009ca0 tx=0x7ff98c02c100 comp rx=0 tx=0).stop 2024-03-17T23:54:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.860+0000 7ff9a2800640 1 -- 172.21.15.12:0/3668368912 shutdown_connections 2024-03-17T23:54:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.860+0000 7ff9a2800640 1 --2- 172.21.15.12:0/3668368912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff99c1055b0 0x7ff99c10be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.860+0000 7ff9a2800640 1 --2- 172.21.15.12:0/3668368912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff99c104bf0 0x7ff99c105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.860+0000 7ff9a2800640 1 --2- 172.21.15.12:0/3668368912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff99c103a00 0x7ff99c103e20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.860+0000 7ff9a2800640 1 -- 172.21.15.12:0/3668368912 >> 172.21.15.12:0/3668368912 conn(0x7ff99c0fef90 msgr2=0x7ff99c1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.860+0000 7ff9a2800640 1 -- 172.21.15.12:0/3668368912 shutdown_connections 2024-03-17T23:54:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.860+0000 7ff9a2800640 1 -- 172.21.15.12:0/3668368912 wait complete. 2024-03-17T23:54:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.862+0000 7ff9a2800640 1 Processor -- start 2024-03-17T23:54:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.862+0000 7ff9a2800640 1 -- start start 2024-03-17T23:54:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.862+0000 7ff9a2800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff99c103a00 0x7ff99c199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.862+0000 7ff9a2800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff99c104bf0 0x7ff99c19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.862+0000 7ff9a2800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff99c1055b0 0x7ff99c1a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.862+0000 7ff9a2800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c10f650 con 0x7ff99c1055b0 2024-03-17T23:54:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.862+0000 7ff9a2800640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c10f4d0 con 0x7ff99c103a00 2024-03-17T23:54:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.862+0000 7ff9a2800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c10f7d0 con 0x7ff99c104bf0 2024-03-17T23:54:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.862+0000 7ff99be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff99c1055b0 0x7ff99c1a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.862+0000 7ff99b400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff99c103a00 0x7ff99c199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.863+0000 7ff992a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff99c104bf0 0x7ff99c19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.863+0000 7ff99b400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff99c103a00 0x7ff99c199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:44472/0 (socket says 172.21.15.12:44472) 2024-03-17T23:54:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.863+0000 7ff99b400640 1 -- 172.21.15.12:0/747160380 learned_addr learned my addr 172.21.15.12:0/747160380 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.863+0000 7ff99b400640 1 -- 172.21.15.12:0/747160380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff99c104bf0 msgr2=0x7ff99c19a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.863+0000 7ff99b400640 1 --2- 172.21.15.12:0/747160380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff99c104bf0 0x7ff99c19a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.863+0000 7ff99b400640 1 -- 172.21.15.12:0/747160380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff99c1055b0 msgr2=0x7ff99c1a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.863+0000 7ff99b400640 1 --2- 172.21.15.12:0/747160380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff99c1055b0 0x7ff99c1a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.863+0000 7ff99b400640 1 -- 172.21.15.12:0/747160380 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff99c1a1960 con 0x7ff99c103a00 2024-03-17T23:54:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.863+0000 7ff99b400640 1 --2- 172.21.15.12:0/747160380 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff99c103a00 0x7ff99c199d40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0324c0 tx=0x7ff98c035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.864+0000 7ff998c00640 1 -- 172.21.15.12:0/747160380 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98c039040 con 0x7ff99c103a00 2024-03-17T23:54:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.864+0000 7ff998c00640 1 -- 172.21.15.12:0/747160380 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff98c044070 con 0x7ff99c103a00 2024-03-17T23:54:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.864+0000 7ff998c00640 1 -- 172.21.15.12:0/747160380 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98c004370 con 0x7ff99c103a00 2024-03-17T23:54:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.864+0000 7ff9a2800640 1 -- 172.21.15.12:0/747160380 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff99c1a1bf0 con 0x7ff99c103a00 2024-03-17T23:54:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.864+0000 7ff9a2800640 1 -- 172.21.15.12:0/747160380 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff99c1a2130 con 0x7ff99c103a00 2024-03-17T23:54:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.865+0000 7ff9a2800640 1 -- 172.21.15.12:0/747160380 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff99c10adf0 con 0x7ff99c103a00 2024-03-17T23:54:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.867+0000 7ff998c00640 1 -- 172.21.15.12:0/747160380 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff98c004510 con 0x7ff99c103a00 2024-03-17T23:54:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.869+0000 7ff998c00640 1 --2- 172.21.15.12:0/747160380 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff9800a2880 0x7ff9800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.869+0000 7ff998c00640 1 -- 172.21.15.12:0/747160380 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff98c0f0b80 con 0x7ff99c103a00 2024-03-17T23:54:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.869+0000 7ff992a00640 1 --2- 172.21.15.12:0/747160380 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff9800a2880 0x7ff9800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.870+0000 7ff992a00640 1 --2- 172.21.15.12:0/747160380 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff9800a2880 0x7ff9800a4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff9840095f0 tx=0x7ff984005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:10.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:10.874+0000 7ff998c00640 1 -- 172.21.15.12:0/747160380 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff98c0be020 con 0x7ff99c103a00 2024-03-17T23:54:10.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:10 smithi012 ceph-mon[31541]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:54:10.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:10 smithi012 ceph-mon[31541]: Saving service mds.all spec with placement smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-03-17T23:54:10.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:10 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:10.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:10 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:54:10.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:10 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:10.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:10 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:10.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:10 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:10.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:10 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:10.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:10 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2711869965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:54:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.078+0000 7ff9a2800640 1 -- 172.21.15.12:0/747160380 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff99c0623e0 con 0x7ff99c103a00 2024-03-17T23:54:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.079+0000 7ff998c00640 1 -- 172.21.15.12:0/747160380 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7ff98c0bd920 con 0x7ff99c103a00 2024-03-17T23:54:11.080 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-17T23:54:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.082+0000 7ff9a2800640 1 -- 172.21.15.12:0/747160380 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff9800a2880 msgr2=0x7ff9800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.082+0000 7ff9a2800640 1 --2- 172.21.15.12:0/747160380 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff9800a2880 0x7ff9800a4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff9840095f0 tx=0x7ff984005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.082+0000 7ff9a2800640 1 -- 172.21.15.12:0/747160380 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff99c103a00 msgr2=0x7ff99c199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.082+0000 7ff9a2800640 1 --2- 172.21.15.12:0/747160380 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff99c103a00 0x7ff99c199d40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0324c0 tx=0x7ff98c035d90 comp rx=0 tx=0).stop 2024-03-17T23:54:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.082+0000 7ff9a2800640 1 -- 172.21.15.12:0/747160380 shutdown_connections 2024-03-17T23:54:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.082+0000 7ff9a2800640 1 --2- 172.21.15.12:0/747160380 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff9800a2880 0x7ff9800a4d40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.082+0000 7ff9a2800640 1 --2- 172.21.15.12:0/747160380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff99c1055b0 0x7ff99c1a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.082+0000 7ff9a2800640 1 --2- 172.21.15.12:0/747160380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff99c104bf0 0x7ff99c19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.082+0000 7ff9a2800640 1 --2- 172.21.15.12:0/747160380 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff99c103a00 0x7ff99c199d40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.082+0000 7ff9a2800640 1 -- 172.21.15.12:0/747160380 >> 172.21.15.12:0/747160380 conn(0x7ff99c0fef90 msgr2=0x7ff99c107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.082+0000 7ff9a2800640 1 -- 172.21.15.12:0/747160380 shutdown_connections 2024-03-17T23:54:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.082+0000 7ff9a2800640 1 -- 172.21.15.12:0/747160380 wait complete. 2024-03-17T23:54:11.103 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:11.207 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:10 smithi191 ceph-mon[36708]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:54:11.207 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:10 smithi191 ceph-mon[36708]: Saving service mds.all spec with placement smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-03-17T23:54:11.207 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:10 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:11.207 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:10 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:54:11.208 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:10 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:11.208 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:10 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:11.208 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:10 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:11.208 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:10 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:11.208 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:10 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2711869965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:54:11.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:10 smithi079 ceph-mon[36241]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:54:11.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:10 smithi079 ceph-mon[36241]: Saving service mds.all spec with placement smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-03-17T23:54:11.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:10 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:11.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:10 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:54:11.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:10 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:11.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:10 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:11.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:10 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:11.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:10 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:11.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:10 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2711869965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:54:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.324+0000 7ffa6ee00640 1 -- 172.21.15.12:0/1839064668 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa681027d0 msgr2=0x7ffa68104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.324+0000 7ffa6ee00640 1 --2- 172.21.15.12:0/1839064668 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa681027d0 0x7ffa68104c40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffa54009860 tx=0x7ffa5402bb70 comp rx=0 tx=0).stop 2024-03-17T23:54:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.324+0000 7ffa6ee00640 1 -- 172.21.15.12:0/1839064668 shutdown_connections 2024-03-17T23:54:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.324+0000 7ffa6ee00640 1 --2- 172.21.15.12:0/1839064668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa68105180 0x7ffa68105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.324+0000 7ffa6ee00640 1 --2- 172.21.15.12:0/1839064668 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa681027d0 0x7ffa68104c40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.324+0000 7ffa6ee00640 1 --2- 172.21.15.12:0/1839064668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa680ffe60 0x7ffa68102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.324+0000 7ffa6ee00640 1 -- 172.21.15.12:0/1839064668 >> 172.21.15.12:0/1839064668 conn(0x7ffa680fba60 msgr2=0x7ffa680fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.324+0000 7ffa6ee00640 1 -- 172.21.15.12:0/1839064668 shutdown_connections 2024-03-17T23:54:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.324+0000 7ffa6ee00640 1 -- 172.21.15.12:0/1839064668 wait complete. 2024-03-17T23:54:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.325+0000 7ffa6ee00640 1 Processor -- start 2024-03-17T23:54:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.325+0000 7ffa6ee00640 1 -- start start 2024-03-17T23:54:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6ee00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa680ffe60 0x7ffa6819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6ee00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa681027d0 0x7ffa6819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6ee00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa68105180 0x7ffa681a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6ee00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffa68113840 con 0x7ffa680ffe60 2024-03-17T23:54:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6ee00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ffa681136c0 con 0x7ffa681027d0 2024-03-17T23:54:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6ee00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ffa681139c0 con 0x7ffa68105180 2024-03-17T23:54:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6da00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa680ffe60 0x7ffa6819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6da00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa680ffe60 0x7ffa6819dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46198/0 (socket says 172.21.15.12:46198) 2024-03-17T23:54:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6d000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa681027d0 0x7ffa6819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6da00640 1 -- 172.21.15.12:0/3581062085 learned_addr learned my addr 172.21.15.12:0/3581062085 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6e400640 1 --2- 172.21.15.12:0/3581062085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa68105180 0x7ffa681a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6d000640 1 -- 172.21.15.12:0/3581062085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa68105180 msgr2=0x7ffa681a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6d000640 1 --2- 172.21.15.12:0/3581062085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa68105180 0x7ffa681a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6d000640 1 -- 172.21.15.12:0/3581062085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa680ffe60 msgr2=0x7ffa6819dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6d000640 1 --2- 172.21.15.12:0/3581062085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa680ffe60 0x7ffa6819dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.326+0000 7ffa6d000640 1 -- 172.21.15.12:0/3581062085 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa681a5ac0 con 0x7ffa681027d0 2024-03-17T23:54:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.327+0000 7ffa6d000640 1 --2- 172.21.15.12:0/3581062085 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa681027d0 0x7ffa6819e3e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffa5402c0b0 tx=0x7ffa54032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.327+0000 7ffa66000640 1 -- 172.21.15.12:0/3581062085 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa54041020 con 0x7ffa681027d0 2024-03-17T23:54:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.327+0000 7ffa66000640 1 -- 172.21.15.12:0/3581062085 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa5402c9f0 con 0x7ffa681027d0 2024-03-17T23:54:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.327+0000 7ffa66000640 1 -- 172.21.15.12:0/3581062085 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa54039040 con 0x7ffa681027d0 2024-03-17T23:54:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.327+0000 7ffa6ee00640 1 -- 172.21.15.12:0/3581062085 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa681a5d50 con 0x7ffa681027d0 2024-03-17T23:54:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.327+0000 7ffa6ee00640 1 -- 172.21.15.12:0/3581062085 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa681a6230 con 0x7ffa681027d0 2024-03-17T23:54:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.328+0000 7ffa6ee00640 1 -- 172.21.15.12:0/3581062085 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa680fd550 con 0x7ffa681027d0 2024-03-17T23:54:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.330+0000 7ffa66000640 1 -- 172.21.15.12:0/3581062085 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ffa54044020 con 0x7ffa681027d0 2024-03-17T23:54:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.331+0000 7ffa66000640 1 --2- 172.21.15.12:0/3581062085 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ffa500a2880 0x7ffa500a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.332+0000 7ffa66000640 1 -- 172.21.15.12:0/3581062085 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ffa540efeb0 con 0x7ffa681027d0 2024-03-17T23:54:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.332+0000 7ffa6da00640 1 --2- 172.21.15.12:0/3581062085 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ffa500a2880 0x7ffa500a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.332+0000 7ffa6da00640 1 --2- 172.21.15.12:0/3581062085 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ffa500a2880 0x7ffa500a4d40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009c90 tx=0x7ffa5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.336+0000 7ffa66000640 1 -- 172.21.15.12:0/3581062085 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffa5404b020 con 0x7ffa681027d0 2024-03-17T23:54:11.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.581+0000 7ffa6ee00640 1 -- 172.21.15.12:0/3581062085 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa68062480 con 0x7ffa681027d0 2024-03-17T23:54:11.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.582+0000 7ffa66000640 1 -- 172.21.15.12:0/3581062085 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7ffa540bd350 con 0x7ffa681027d0 2024-03-17T23:54:11.583 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:11.583 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-17T23:54:11.583 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 1 2024-03-17T23:54:11.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.585+0000 7ffa6ee00640 1 -- 172.21.15.12:0/3581062085 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ffa500a2880 msgr2=0x7ffa500a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:11.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.585+0000 7ffa6ee00640 1 --2- 172.21.15.12:0/3581062085 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ffa500a2880 0x7ffa500a4d40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009c90 tx=0x7ffa5c005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:11.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.585+0000 7ffa6ee00640 1 -- 172.21.15.12:0/3581062085 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa681027d0 msgr2=0x7ffa6819e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:11.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.585+0000 7ffa6ee00640 1 --2- 172.21.15.12:0/3581062085 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa681027d0 0x7ffa6819e3e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffa5402c0b0 tx=0x7ffa54032430 comp rx=0 tx=0).stop 2024-03-17T23:54:11.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.585+0000 7ffa6ee00640 1 -- 172.21.15.12:0/3581062085 shutdown_connections 2024-03-17T23:54:11.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.585+0000 7ffa6ee00640 1 --2- 172.21.15.12:0/3581062085 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ffa500a2880 0x7ffa500a4d40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:11.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.585+0000 7ffa6ee00640 1 --2- 172.21.15.12:0/3581062085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa68105180 0x7ffa681a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:11.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.585+0000 7ffa6ee00640 1 --2- 172.21.15.12:0/3581062085 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa681027d0 0x7ffa6819e3e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:11.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.585+0000 7ffa6ee00640 1 --2- 172.21.15.12:0/3581062085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffa680ffe60 0x7ffa6819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:11.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.585+0000 7ffa6ee00640 1 -- 172.21.15.12:0/3581062085 >> 172.21.15.12:0/3581062085 conn(0x7ffa680fba60 msgr2=0x7ffa68100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:11.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.585+0000 7ffa6ee00640 1 -- 172.21.15.12:0/3581062085 shutdown_connections 2024-03-17T23:54:11.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:11.585+0000 7ffa6ee00640 1 -- 172.21.15.12:0/3581062085 wait complete. 2024-03-17T23:54:12.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:11 smithi012 ceph-mon[31541]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:12.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:11 smithi012 ceph-mon[31541]: Deploying daemon mds.l on smithi191 2024-03-17T23:54:12.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:11 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/747160380' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:54:12.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:11 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3581062085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:12.182 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:11 smithi191 ceph-mon[36708]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:12.182 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:11 smithi191 ceph-mon[36708]: Deploying daemon mds.l on smithi191 2024-03-17T23:54:12.183 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:11 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/747160380' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:54:12.183 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:11 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3581062085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:12.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:11 smithi079 ceph-mon[36241]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:12.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:11 smithi079 ceph-mon[36241]: Deploying daemon mds.l on smithi191 2024-03-17T23:54:12.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:11 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/747160380' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:54:12.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:11 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3581062085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:12.576 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 17 23:54:12 smithi191 systemd[1]: Starting Ceph mds.l for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-17T23:54:12.841 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 17 23:54:12 smithi191 podman[66875]: 2024-03-17 23:54:12.695364507 +0000 UTC m=+0.017797617 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-17T23:54:12.841 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 17 23:54:12 smithi191 podman[66875]: 2024-03-17 23:54:12.841354767 +0000 UTC m=+0.163787858 container create b100e76a6ff3149cfd49771a3f27aa79be85182835834dba13be31678ab8cccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-l, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-17T23:54:13.091 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 17 23:54:13 smithi191 podman[66875]: 2024-03-17 23:54:13.089198522 +0000 UTC m=+0.411631626 container init b100e76a6ff3149cfd49771a3f27aa79be85182835834dba13be31678ab8cccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-l, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-17T23:54:13.591 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 17 23:54:13 smithi191 podman[66875]: 2024-03-17 23:54:13.091960579 +0000 UTC m=+0.414393670 container start b100e76a6ff3149cfd49771a3f27aa79be85182835834dba13be31678ab8cccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-l, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-17T23:54:13.591 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 17 23:54:13 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-l[66890]: starting mds.l at 2024-03-17T23:54:13.591 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 17 23:54:13 smithi191 bash[66875]: b100e76a6ff3149cfd49771a3f27aa79be85182835834dba13be31678ab8cccc 2024-03-17T23:54:13.591 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 17 23:54:13 smithi191 systemd[1]: Started Ceph mds.l for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:54:13.606 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.841+0000 7f898da00640 1 -- 172.21.15.12:0/2584632227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89880773e0 msgr2=0x7f8988077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.841+0000 7f898da00640 1 --2- 172.21.15.12:0/2584632227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89880773e0 0x7f8988077840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8970009900 tx=0x7f897002bc30 comp rx=0 tx=0).stop 2024-03-17T23:54:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.841+0000 7f898da00640 1 -- 172.21.15.12:0/2584632227 shutdown_connections 2024-03-17T23:54:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.841+0000 7f898da00640 1 --2- 172.21.15.12:0/2584632227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8988077d80 0x7f8988078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.841+0000 7f898da00640 1 --2- 172.21.15.12:0/2584632227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89880773e0 0x7f8988077840 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.841+0000 7f898da00640 1 --2- 172.21.15.12:0/2584632227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89880761a0 0x7f89880765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.841+0000 7f898da00640 1 -- 172.21.15.12:0/2584632227 >> 172.21.15.12:0/2584632227 conn(0x7f89880fef90 msgr2=0x7f89881013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.842+0000 7f898da00640 1 -- 172.21.15.12:0/2584632227 shutdown_connections 2024-03-17T23:54:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.842+0000 7f898da00640 1 -- 172.21.15.12:0/2584632227 wait complete. 2024-03-17T23:54:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.843+0000 7f898da00640 1 Processor -- start 2024-03-17T23:54:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.843+0000 7f898da00640 1 -- start start 2024-03-17T23:54:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.843+0000 7f898da00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89880761a0 0x7f898819e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.843+0000 7f898da00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89880773e0 0x7f898819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.843+0000 7f898da00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8988077d80 0x7f89881a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.843+0000 7f898da00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8988113ad0 con 0x7f89880761a0 2024-03-17T23:54:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.843+0000 7f898da00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8988113950 con 0x7f89880773e0 2024-03-17T23:54:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.843+0000 7f898da00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8988113c50 con 0x7f8988077d80 2024-03-17T23:54:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.844+0000 7f8986a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89880761a0 0x7f898819e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.844+0000 7f8986000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89880773e0 0x7f898819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.844+0000 7f8986a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89880761a0 0x7f898819e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60418/0 (socket says 172.21.15.12:60418) 2024-03-17T23:54:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.844+0000 7f8986a00640 1 -- 172.21.15.12:0/224131424 learned_addr learned my addr 172.21.15.12:0/224131424 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.844+0000 7f8987400640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8988077d80 0x7f89881a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.844+0000 7f8986a00640 1 -- 172.21.15.12:0/224131424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8988077d80 msgr2=0x7f89881a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.844+0000 7f8986a00640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8988077d80 0x7f89881a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.844+0000 7f8986a00640 1 -- 172.21.15.12:0/224131424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89880773e0 msgr2=0x7f898819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.844+0000 7f8986a00640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89880773e0 0x7f898819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.844+0000 7f8986a00640 1 -- 172.21.15.12:0/224131424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89881a5cf0 con 0x7f89880761a0 2024-03-17T23:54:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.844+0000 7f8987400640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8988077d80 0x7f89881a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:54:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.844+0000 7f8986000640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89880773e0 0x7f898819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.844+0000 7f8986a00640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89880761a0 0x7f898819e120 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f898000ebc0 tx=0x7f898000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.845+0000 7f897f400640 1 -- 172.21.15.12:0/224131424 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8980010630 con 0x7f89880761a0 2024-03-17T23:54:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.845+0000 7f897f400640 1 -- 172.21.15.12:0/224131424 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8980010cb0 con 0x7f89880761a0 2024-03-17T23:54:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.845+0000 7f897f400640 1 -- 172.21.15.12:0/224131424 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8980018bd0 con 0x7f89880761a0 2024-03-17T23:54:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.845+0000 7f898da00640 1 -- 172.21.15.12:0/224131424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89881a5fe0 con 0x7f89880761a0 2024-03-17T23:54:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.845+0000 7f898da00640 1 -- 172.21.15.12:0/224131424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89881a63f0 con 0x7f89880761a0 2024-03-17T23:54:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.846+0000 7f898da00640 1 -- 172.21.15.12:0/224131424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8988110370 con 0x7f89880761a0 2024-03-17T23:54:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.848+0000 7f897f400640 1 -- 172.21.15.12:0/224131424 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f89800107d0 con 0x7f89880761a0 2024-03-17T23:54:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.850+0000 7f897f400640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f896c0a2880 0x7f896c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.850+0000 7f897f400640 1 -- 172.21.15.12:0/224131424 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f89800cec70 con 0x7f89880761a0 2024-03-17T23:54:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.850+0000 7f8986000640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f896c0a2880 0x7f896c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.851+0000 7f8986000640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f896c0a2880 0x7f896c0a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f897002c240 tx=0x7f897002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:13.855+0000 7f897f400640 1 -- 172.21.15.12:0/224131424 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f898009c110 con 0x7f89880761a0 2024-03-17T23:54:14.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.101+0000 7f898da00640 1 -- 172.21.15.12:0/224131424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f898802d460 con 0x7f89880761a0 2024-03-17T23:54:14.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.102+0000 7f897f400640 1 -- 172.21.15.12:0/224131424 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f898009ba10 con 0x7f89880761a0 2024-03-17T23:54:14.103 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:14.103 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-17T23:54:14.103 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 2 2024-03-17T23:54:14.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.105+0000 7f898da00640 1 -- 172.21.15.12:0/224131424 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f896c0a2880 msgr2=0x7f896c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:14.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.105+0000 7f898da00640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f896c0a2880 0x7f896c0a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f897002c240 tx=0x7f897002df90 comp rx=0 tx=0).stop 2024-03-17T23:54:14.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.105+0000 7f898da00640 1 -- 172.21.15.12:0/224131424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89880761a0 msgr2=0x7f898819e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:14.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.105+0000 7f898da00640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89880761a0 0x7f898819e120 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f898000ebc0 tx=0x7f898000c6a0 comp rx=0 tx=0).stop 2024-03-17T23:54:14.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.105+0000 7f898da00640 1 -- 172.21.15.12:0/224131424 shutdown_connections 2024-03-17T23:54:14.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.105+0000 7f898da00640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f896c0a2880 0x7f896c0a4d40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:14.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.105+0000 7f898da00640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8988077d80 0x7f89881a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:14.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.105+0000 7f898da00640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f89880773e0 0x7f898819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:14.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.105+0000 7f898da00640 1 --2- 172.21.15.12:0/224131424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89880761a0 0x7f898819e120 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:14.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.105+0000 7f898da00640 1 -- 172.21.15.12:0/224131424 >> 172.21.15.12:0/224131424 conn(0x7f89880fef90 msgr2=0x7f8988100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:14.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.106+0000 7f898da00640 1 -- 172.21.15.12:0/224131424 shutdown_connections 2024-03-17T23:54:14.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:14.106+0000 7f898da00640 1 -- 172.21.15.12:0/224131424 wait complete. 2024-03-17T23:54:14.123 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:13 smithi012 ceph-mon[31541]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:14.123 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:13 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.191:6832/1787103083,v1:172.21.15.191:6833/1787103083] up:boot 2024-03-17T23:54:14.123 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:13 smithi012 ceph-mon[31541]: fsmap 1 up:standby 2024-03-17T23:54:14.123 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-17T23:54:14.123 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:14.123 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:14.123 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:14.123 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:14.123 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:14.123 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:14.194 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:13 smithi079 ceph-mon[36241]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:14.194 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:13 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.191:6832/1787103083,v1:172.21.15.191:6833/1787103083] up:boot 2024-03-17T23:54:14.194 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:13 smithi079 ceph-mon[36241]: fsmap 1 up:standby 2024-03-17T23:54:14.194 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:13 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-17T23:54:14.194 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:13 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:14.194 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:13 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:14.194 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:13 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:14.194 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:13 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:14.195 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:13 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:14.195 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:13 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:14.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:13 smithi191 ceph-mon[36708]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:14.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:13 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.191:6832/1787103083,v1:172.21.15.191:6833/1787103083] up:boot 2024-03-17T23:54:14.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:13 smithi191 ceph-mon[36708]: fsmap 1 up:standby 2024-03-17T23:54:14.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:13 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-17T23:54:14.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:13 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:14.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:13 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:14.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:13 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:14.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:13 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:14.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:13 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:14.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:13 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:15.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:14 smithi079 ceph-mon[36241]: Deploying daemon mds.k on smithi079 2024-03-17T23:54:15.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:14 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/224131424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:15.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:14 smithi012 ceph-mon[31541]: Deploying daemon mds.k on smithi079 2024-03-17T23:54:15.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:14 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/224131424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:15.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:14 smithi191 ceph-mon[36708]: Deploying daemon mds.k on smithi079 2024-03-17T23:54:15.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:14 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/224131424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:15.796 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 17 23:54:15 smithi079 systemd[1]: Starting Ceph mds.k for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-17T23:54:16.128 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:16.163 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:15 smithi079 ceph-mon[36241]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:16.163 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 17 23:54:15 smithi079 podman[65322]: 2024-03-17 23:54:15.793403521 +0000 UTC m=+0.018228735 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-17T23:54:16.163 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 17 23:54:15 smithi079 podman[65322]: 2024-03-17 23:54:15.900630531 +0000 UTC m=+0.125455754 container create 08729b28bcffeea94e32916fd3df8a67ce5e69c3dc0678055a6b4f6e9cfef551 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-k, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-17T23:54:16.163 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 17 23:54:16 smithi079 podman[65322]: 2024-03-17 23:54:16.136701476 +0000 UTC m=+0.361526679 container init 08729b28bcffeea94e32916fd3df8a67ce5e69c3dc0678055a6b4f6e9cfef551 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-k, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-17T23:54:16.164 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 17 23:54:16 smithi079 podman[65322]: 2024-03-17 23:54:16.140139522 +0000 UTC m=+0.364964735 container start 08729b28bcffeea94e32916fd3df8a67ce5e69c3dc0678055a6b4f6e9cfef551 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-17T23:54:16.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:15 smithi012 ceph-mon[31541]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:16.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:15 smithi191 ceph-mon[36708]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:16.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.346+0000 7fa350e00640 1 -- 172.21.15.12:0/1249930811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa34c1029e0 msgr2=0x7fa34c102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:16.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.346+0000 7fa350e00640 1 --2- 172.21.15.12:0/1249930811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa34c1029e0 0x7fa34c102e60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa338009900 tx=0x7fa33802bc40 comp rx=0 tx=0).stop 2024-03-17T23:54:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.346+0000 7fa350e00640 1 -- 172.21.15.12:0/1249930811 shutdown_connections 2024-03-17T23:54:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.346+0000 7fa350e00640 1 --2- 172.21.15.12:0/1249930811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa34c1033a0 0x7fa34c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.346+0000 7fa350e00640 1 --2- 172.21.15.12:0/1249930811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa34c1029e0 0x7fa34c102e60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.346+0000 7fa350e00640 1 --2- 172.21.15.12:0/1249930811 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa34c1017f0 0x7fa34c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.346+0000 7fa350e00640 1 -- 172.21.15.12:0/1249930811 >> 172.21.15.12:0/1249930811 conn(0x7fa34c0fcd80 msgr2=0x7fa34c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.346+0000 7fa350e00640 1 -- 172.21.15.12:0/1249930811 shutdown_connections 2024-03-17T23:54:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.346+0000 7fa350e00640 1 -- 172.21.15.12:0/1249930811 wait complete. 2024-03-17T23:54:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.348+0000 7fa350e00640 1 Processor -- start 2024-03-17T23:54:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.348+0000 7fa350e00640 1 -- start start 2024-03-17T23:54:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.348+0000 7fa350e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa34c1017f0 0x7fa34c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.348+0000 7fa350e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa34c1029e0 0x7fa34c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.348+0000 7fa350e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa34c1033a0 0x7fa34c19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.348+0000 7fa350e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c10d4c0 con 0x7fa34c1017f0 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.348+0000 7fa350e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c10d340 con 0x7fa34c1029e0 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.348+0000 7fa350e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c10d640 con 0x7fa34c1033a0 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.348+0000 7fa349e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa34c1017f0 0x7fa34c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.348+0000 7fa349400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa34c1029e0 0x7fa34c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.348+0000 7fa349e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa34c1017f0 0x7fa34c197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60430/0 (socket says 172.21.15.12:60430) 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.348+0000 7fa349400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa34c1029e0 0x7fa34c198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:54902/0 (socket says 172.21.15.12:54902) 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.348+0000 7fa349e00640 1 -- 172.21.15.12:0/2983128748 learned_addr learned my addr 172.21.15.12:0/2983128748 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.349+0000 7fa34a800640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa34c1033a0 0x7fa34c19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.349+0000 7fa349e00640 1 -- 172.21.15.12:0/2983128748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa34c1033a0 msgr2=0x7fa34c19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.349+0000 7fa349e00640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa34c1033a0 0x7fa34c19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.349+0000 7fa349e00640 1 -- 172.21.15.12:0/2983128748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa34c1029e0 msgr2=0x7fa34c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.349+0000 7fa349e00640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa34c1029e0 0x7fa34c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.349+0000 7fa349e00640 1 -- 172.21.15.12:0/2983128748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa34c19f700 con 0x7fa34c1017f0 2024-03-17T23:54:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.349+0000 7fa349400640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa34c1029e0 0x7fa34c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.349+0000 7fa349e00640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa34c1017f0 0x7fa34c197ae0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00e9c0 tx=0x7fa33c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:16.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.352+0000 7fa342a00640 1 -- 172.21.15.12:0/2983128748 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c00cd80 con 0x7fa34c1017f0 2024-03-17T23:54:16.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.352+0000 7fa342a00640 1 -- 172.21.15.12:0/2983128748 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa33c0035f0 con 0x7fa34c1017f0 2024-03-17T23:54:16.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.352+0000 7fa342a00640 1 -- 172.21.15.12:0/2983128748 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c010690 con 0x7fa34c1017f0 2024-03-17T23:54:16.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.352+0000 7fa350e00640 1 -- 172.21.15.12:0/2983128748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa34c19f9f0 con 0x7fa34c1017f0 2024-03-17T23:54:16.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.352+0000 7fa350e00640 1 -- 172.21.15.12:0/2983128748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa34c19ff10 con 0x7fa34c1017f0 2024-03-17T23:54:16.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.353+0000 7fa350e00640 1 -- 172.21.15.12:0/2983128748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa34c108b10 con 0x7fa34c1017f0 2024-03-17T23:54:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.355+0000 7fa342a00640 1 -- 172.21.15.12:0/2983128748 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa33c0037a0 con 0x7fa34c1017f0 2024-03-17T23:54:16.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.357+0000 7fa342a00640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3300a2880 0x7fa3300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:16.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.357+0000 7fa342a00640 1 -- 172.21.15.12:0/2983128748 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa33c0ce9e0 con 0x7fa34c1017f0 2024-03-17T23:54:16.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.358+0000 7fa349400640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3300a2880 0x7fa3300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:16.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.359+0000 7fa349400640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3300a2880 0x7fa3300a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa33802c250 tx=0x7fa33802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:16.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.362+0000 7fa342a00640 1 -- 172.21.15.12:0/2983128748 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa33c09be80 con 0x7fa34c1017f0 2024-03-17T23:54:16.447 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 17 23:54:16 smithi079 ceph-mds[65342]: -- 172.21.15.79:0/48054360 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55638ae41a40 con 0x55638ae15000 2024-03-17T23:54:16.448 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 17 23:54:16 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-k[65337]: starting mds.k at 2024-03-17T23:54:16.448 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 17 23:54:16 smithi079 bash[65322]: 08729b28bcffeea94e32916fd3df8a67ce5e69c3dc0678055a6b4f6e9cfef551 2024-03-17T23:54:16.448 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 17 23:54:16 smithi079 systemd[1]: Started Ceph mds.k for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:54:16.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.609+0000 7fa350e00640 1 -- 172.21.15.12:0/2983128748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa34c0624a0 con 0x7fa34c1017f0 2024-03-17T23:54:16.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.611+0000 7fa342a00640 1 -- 172.21.15.12:0/2983128748 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fa33c09b780 con 0x7fa34c1017f0 2024-03-17T23:54:16.611 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:16.611 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-17T23:54:16.612 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 2 2024-03-17T23:54:16.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.613+0000 7fa350e00640 1 -- 172.21.15.12:0/2983128748 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3300a2880 msgr2=0x7fa3300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:16.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.613+0000 7fa350e00640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3300a2880 0x7fa3300a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa33802c250 tx=0x7fa33802cf90 comp rx=0 tx=0).stop 2024-03-17T23:54:16.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.613+0000 7fa350e00640 1 -- 172.21.15.12:0/2983128748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa34c1017f0 msgr2=0x7fa34c197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:16.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.613+0000 7fa350e00640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa34c1017f0 0x7fa34c197ae0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00e9c0 tx=0x7fa33c00ee80 comp rx=0 tx=0).stop 2024-03-17T23:54:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.613+0000 7fa350e00640 1 -- 172.21.15.12:0/2983128748 shutdown_connections 2024-03-17T23:54:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.613+0000 7fa350e00640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3300a2880 0x7fa3300a4d40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.613+0000 7fa350e00640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa34c1033a0 0x7fa34c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.613+0000 7fa350e00640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa34c1029e0 0x7fa34c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.613+0000 7fa350e00640 1 --2- 172.21.15.12:0/2983128748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa34c1017f0 0x7fa34c197ae0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.613+0000 7fa350e00640 1 -- 172.21.15.12:0/2983128748 >> 172.21.15.12:0/2983128748 conn(0x7fa34c0fcd80 msgr2=0x7fa34c105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.614+0000 7fa350e00640 1 -- 172.21.15.12:0/2983128748 shutdown_connections 2024-03-17T23:54:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:16.614+0000 7fa350e00640 1 -- 172.21.15.12:0/2983128748 wait complete. 2024-03-17T23:54:17.466 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:17 smithi079 ceph-mon[36241]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:17.466 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:17.466 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:17.466 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:17.466 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:17.466 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:17.466 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:17.466 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:17 smithi079 ceph-mon[36241]: Deploying daemon mds.j on smithi012 2024-03-17T23:54:17.466 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:17 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2983128748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:17.466 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:17 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.79:6832/843208408,v1:172.21.15.79:6833/843208408] up:boot 2024-03-17T23:54:17.467 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:17 smithi079 ceph-mon[36241]: fsmap 2 up:standby 2024-03-17T23:54:17.467 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:17 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-17T23:54:17.564 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:17 smithi012 ceph-mon[31541]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:17.564 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:17.565 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:17.565 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:17.565 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:17.565 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:17.565 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:17.565 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:17 smithi012 ceph-mon[31541]: Deploying daemon mds.j on smithi012 2024-03-17T23:54:17.565 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:17 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2983128748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:17.565 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:17 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.79:6832/843208408,v1:172.21.15.79:6833/843208408] up:boot 2024-03-17T23:54:17.565 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:17 smithi012 ceph-mon[31541]: fsmap 2 up:standby 2024-03-17T23:54:17.565 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:17 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-17T23:54:17.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:17 smithi191 ceph-mon[36708]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:17.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:17.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:17.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:17.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:17.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:17.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:17.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:17 smithi191 ceph-mon[36708]: Deploying daemon mds.j on smithi012 2024-03-17T23:54:17.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:17 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2983128748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:17.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:17 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.79:6832/843208408,v1:172.21.15.79:6833/843208408] up:boot 2024-03-17T23:54:17.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:17 smithi191 ceph-mon[36708]: fsmap 2 up:standby 2024-03-17T23:54:17.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:17 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-17T23:54:18.634 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:19.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.073+0000 7fbe72000640 1 -- 172.21.15.12:0/488694435 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbe6c1089c0 msgr2=0x7fbe6c10adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:19.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.073+0000 7fbe72000640 1 --2- 172.21.15.12:0/488694435 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbe6c1089c0 0x7fbe6c10adb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c009900 tx=0x7fbe5c02bc40 comp rx=0 tx=0).stop 2024-03-17T23:54:19.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.076+0000 7fbe72000640 1 -- 172.21.15.12:0/488694435 shutdown_connections 2024-03-17T23:54:19.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.076+0000 7fbe72000640 1 --2- 172.21.15.12:0/488694435 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe6c10de80 0x7fbe6c110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:19.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.076+0000 7fbe72000640 1 --2- 172.21.15.12:0/488694435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe6c10b380 0x7fbe6c10d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:19.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.076+0000 7fbe72000640 1 --2- 172.21.15.12:0/488694435 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbe6c1089c0 0x7fbe6c10adb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:19.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.076+0000 7fbe72000640 1 -- 172.21.15.12:0/488694435 >> 172.21.15.12:0/488694435 conn(0x7fbe6c06e640 msgr2=0x7fbe6c070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:19.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.078+0000 7fbe72000640 1 -- 172.21.15.12:0/488694435 shutdown_connections 2024-03-17T23:54:19.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.082+0000 7fbe72000640 1 -- 172.21.15.12:0/488694435 wait complete. 2024-03-17T23:54:19.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.083+0000 7fbe72000640 1 Processor -- start 2024-03-17T23:54:19.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.083+0000 7fbe72000640 1 -- start start 2024-03-17T23:54:19.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.084+0000 7fbe72000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe6c1089c0 0x7fbe6c19cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:19.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.084+0000 7fbe72000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbe6c10b380 0x7fbe6c19d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:19.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.084+0000 7fbe72000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe6c10de80 0x7fbe6c199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:19.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.084+0000 7fbe72000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c113ab0 con 0x7fbe6c1089c0 2024-03-17T23:54:19.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.084+0000 7fbe72000640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c113930 con 0x7fbe6c10b380 2024-03-17T23:54:19.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.084+0000 7fbe72000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c113c30 con 0x7fbe6c10de80 2024-03-17T23:54:19.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.084+0000 7fbe6aa00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbe6c10b380 0x7fbe6c19d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:19.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.084+0000 7fbe6aa00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbe6c10b380 0x7fbe6c19d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:54920/0 (socket says 172.21.15.12:54920) 2024-03-17T23:54:19.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.084+0000 7fbe6aa00640 1 -- 172.21.15.12:0/981489160 learned_addr learned my addr 172.21.15.12:0/981489160 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:19.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.084+0000 7fbe6be00640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe6c10de80 0x7fbe6c199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:19.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.085+0000 7fbe6b400640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe6c1089c0 0x7fbe6c19cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:19.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.085+0000 7fbe6be00640 1 -- 172.21.15.12:0/981489160 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbe6c10b380 msgr2=0x7fbe6c19d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:19.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.085+0000 7fbe6be00640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbe6c10b380 0x7fbe6c19d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:19.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.085+0000 7fbe6be00640 1 -- 172.21.15.12:0/981489160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe6c1089c0 msgr2=0x7fbe6c19cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:19.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.085+0000 7fbe6be00640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe6c1089c0 0x7fbe6c19cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:19.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.085+0000 7fbe6be00640 1 -- 172.21.15.12:0/981489160 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe6c199ef0 con 0x7fbe6c10de80 2024-03-17T23:54:19.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.085+0000 7fbe6b400640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe6c1089c0 0x7fbe6c19cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:54:19.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.086+0000 7fbe6be00640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe6c10de80 0x7fbe6c199630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbe6400cab0 tx=0x7fbe6400cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:19.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.088+0000 7fbe63e00640 1 -- 172.21.15.12:0/981489160 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6400ae20 con 0x7fbe6c10de80 2024-03-17T23:54:19.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.088+0000 7fbe63e00640 1 -- 172.21.15.12:0/981489160 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe6400f930 con 0x7fbe6c10de80 2024-03-17T23:54:19.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.088+0000 7fbe63e00640 1 -- 172.21.15.12:0/981489160 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe64018b30 con 0x7fbe6c10de80 2024-03-17T23:54:19.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.088+0000 7fbe72000640 1 -- 172.21.15.12:0/981489160 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe6c1ae940 con 0x7fbe6c10de80 2024-03-17T23:54:19.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.088+0000 7fbe72000640 1 -- 172.21.15.12:0/981489160 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe6c1aed50 con 0x7fbe6c10de80 2024-03-17T23:54:19.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.090+0000 7fbe72000640 1 -- 172.21.15.12:0/981489160 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe6c1af040 con 0x7fbe6c10de80 2024-03-17T23:54:19.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.092+0000 7fbe63e00640 1 -- 172.21.15.12:0/981489160 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbe6400fae0 con 0x7fbe6c10de80 2024-03-17T23:54:19.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.093+0000 7fbe63e00640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe500a28d0 0x7fbe500a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:19.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.093+0000 7fbe63e00640 1 -- 172.21.15.12:0/981489160 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbe640cee80 con 0x7fbe6c10de80 2024-03-17T23:54:19.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.093+0000 7fbe6b400640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe500a28d0 0x7fbe500a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:19.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.094+0000 7fbe6b400640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe500a28d0 0x7fbe500a4d90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c009850 tx=0x7fbe5c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:19.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.096+0000 7fbe63e00640 1 -- 172.21.15.12:0/981489160 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbe64014030 con 0x7fbe6c10de80 2024-03-17T23:54:19.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.343+0000 7fbe72000640 1 -- 172.21.15.12:0/981489160 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe6c1af210 con 0x7fbe6c10de80 2024-03-17T23:54:19.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.344+0000 7fbe63e00640 1 -- 172.21.15.12:0/981489160 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7fbe6409c3a0 con 0x7fbe6c10de80 2024-03-17T23:54:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:19.345 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-17T23:54:19.345 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 3 2024-03-17T23:54:19.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.347+0000 7fbe72000640 1 -- 172.21.15.12:0/981489160 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe500a28d0 msgr2=0x7fbe500a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:19.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.347+0000 7fbe72000640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe500a28d0 0x7fbe500a4d90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c009850 tx=0x7fbe5c005fd0 comp rx=0 tx=0).stop 2024-03-17T23:54:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.347+0000 7fbe72000640 1 -- 172.21.15.12:0/981489160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe6c10de80 msgr2=0x7fbe6c199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.347+0000 7fbe72000640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe6c10de80 0x7fbe6c199630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbe6400cab0 tx=0x7fbe6400cf70 comp rx=0 tx=0).stop 2024-03-17T23:54:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.347+0000 7fbe72000640 1 -- 172.21.15.12:0/981489160 shutdown_connections 2024-03-17T23:54:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.347+0000 7fbe72000640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbe500a28d0 0x7fbe500a4d90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.347+0000 7fbe72000640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe6c10de80 0x7fbe6c199630 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.347+0000 7fbe72000640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbe6c10b380 0x7fbe6c19d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.347+0000 7fbe72000640 1 --2- 172.21.15.12:0/981489160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe6c1089c0 0x7fbe6c19cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.347+0000 7fbe72000640 1 -- 172.21.15.12:0/981489160 >> 172.21.15.12:0/981489160 conn(0x7fbe6c06e640 msgr2=0x7fbe6c10e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.347+0000 7fbe72000640 1 -- 172.21.15.12:0/981489160 shutdown_connections 2024-03-17T23:54:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:19.347+0000 7fbe72000640 1 -- 172.21.15.12:0/981489160 wait complete. 2024-03-17T23:54:19.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:19 smithi191 ceph-mon[36708]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:19.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:19 smithi012 ceph-mon[31541]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:19.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:19 smithi079 ceph-mon[36241]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:19.988 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 17 23:54:19 smithi012 podman[69096]: 2024-03-17 23:54:19.700248711 +0000 UTC m=+0.155803259 container create 07026a04c1a6828048de07e93ebfa8c14be36657946b74c9e5ed127e6571fea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-j, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-17T23:54:19.988 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 17 23:54:19 smithi012 podman[69096]: 2024-03-17 23:54:19.987347677 +0000 UTC m=+0.442902205 container init 07026a04c1a6828048de07e93ebfa8c14be36657946b74c9e5ed127e6571fea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-17T23:54:20.322 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 17 23:54:19 smithi012 podman[69096]: 2024-03-17 23:54:19.997497424 +0000 UTC m=+0.453051918 container start 07026a04c1a6828048de07e93ebfa8c14be36657946b74c9e5ed127e6571fea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True) 2024-03-17T23:54:20.322 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 17 23:54:20 smithi012 ceph-mds[69115]: -- 172.21.15.12:0/1878567173 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5627b42ffa40 con 0x5627b42d2800 2024-03-17T23:54:20.322 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 17 23:54:20 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-j[69111]: starting mds.j at 2024-03-17T23:54:20.322 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 17 23:54:20 smithi012 bash[69096]: 07026a04c1a6828048de07e93ebfa8c14be36657946b74c9e5ed127e6571fea4 2024-03-17T23:54:20.322 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 17 23:54:20 smithi012 systemd[1]: Started Ceph mds.j for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:54:20.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:20 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/981489160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:20.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:20 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:20.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:20 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:20.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:20 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:20.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:20 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:20.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:20 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:20.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:20 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:20.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:20 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/981489160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:20.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:20 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:20.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:20 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:20.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:20 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:20.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:20 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:20.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:20 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:20.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:20 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:20.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:20 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/981489160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:20.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:20 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:20.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:20 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:20.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:20 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:20.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:20 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:20.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:20 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:20.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:20 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:21.367 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:21.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:21 smithi191 ceph-mon[36708]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:21.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:21 smithi191 ceph-mon[36708]: Deploying daemon mds.d on smithi012 2024-03-17T23:54:21.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:21 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.12:6834/1091924746,v1:172.21.15.12:6835/1091924746] up:boot 2024-03-17T23:54:21.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:21 smithi191 ceph-mon[36708]: fsmap 3 up:standby 2024-03-17T23:54:21.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:21 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-17T23:54:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.610+0000 7f83ab800640 1 -- 172.21.15.12:0/3449850949 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83a4077d80 msgr2=0x7f83a4078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.610+0000 7f83ab800640 1 --2- 172.21.15.12:0/3449850949 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83a4077d80 0x7f83a4078620 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f839c009900 tx=0x7f839c02bc30 comp rx=0 tx=0).stop 2024-03-17T23:54:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.610+0000 7f83ab800640 1 -- 172.21.15.12:0/3449850949 shutdown_connections 2024-03-17T23:54:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.610+0000 7f83ab800640 1 --2- 172.21.15.12:0/3449850949 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83a4077d80 0x7f83a4078620 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.610+0000 7f83ab800640 1 --2- 172.21.15.12:0/3449850949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83a40773e0 0x7f83a4077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.610+0000 7f83ab800640 1 --2- 172.21.15.12:0/3449850949 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f83a40761a0 0x7f83a40765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.610+0000 7f83ab800640 1 -- 172.21.15.12:0/3449850949 >> 172.21.15.12:0/3449850949 conn(0x7f83a40fef90 msgr2=0x7f83a41013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.610+0000 7f83ab800640 1 -- 172.21.15.12:0/3449850949 shutdown_connections 2024-03-17T23:54:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.610+0000 7f83ab800640 1 -- 172.21.15.12:0/3449850949 wait complete. 2024-03-17T23:54:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.611+0000 7f83ab800640 1 Processor -- start 2024-03-17T23:54:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.611+0000 7f83ab800640 1 -- start start 2024-03-17T23:54:21.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.611+0000 7f83ab800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f83a40761a0 0x7f83a419e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.611+0000 7f83ab800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83a40773e0 0x7f83a419e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.611+0000 7f83ab800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83a4077d80 0x7f83a41a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.611+0000 7f83ab800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f83a4113a90 con 0x7f83a40773e0 2024-03-17T23:54:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.611+0000 7f83ab800640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f83a4113910 con 0x7f83a40761a0 2024-03-17T23:54:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.611+0000 7f83ab800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f83a4113c10 con 0x7f83a4077d80 2024-03-17T23:54:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.612+0000 7f839be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83a40773e0 0x7f83a419e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.612+0000 7f83a9800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83a4077d80 0x7f83a41a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.612+0000 7f83a9800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83a4077d80 0x7f83a41a5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53782/0 (socket says 172.21.15.12:53782) 2024-03-17T23:54:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.612+0000 7f83a9800640 1 -- 172.21.15.12:0/1837097515 learned_addr learned my addr 172.21.15.12:0/1837097515 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.612+0000 7f83a8e00640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f83a40761a0 0x7f83a419e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.612+0000 7f839be00640 1 -- 172.21.15.12:0/1837097515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83a4077d80 msgr2=0x7f83a41a5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.612+0000 7f839be00640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83a4077d80 0x7f83a41a5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.612+0000 7f839be00640 1 -- 172.21.15.12:0/1837097515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f83a40761a0 msgr2=0x7f83a419e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.612+0000 7f839be00640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f83a40761a0 0x7f83a419e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.612+0000 7f839be00640 1 -- 172.21.15.12:0/1837097515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a41a5ca0 con 0x7f83a40773e0 2024-03-17T23:54:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.612+0000 7f83a8e00640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f83a40761a0 0x7f83a419e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.612+0000 7f839be00640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83a40773e0 0x7f83a419e5c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f838c00e9f0 tx=0x7f838c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.613+0000 7f83a2000640 1 -- 172.21.15.12:0/1837097515 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c00cd80 con 0x7f83a40773e0 2024-03-17T23:54:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.613+0000 7f83ab800640 1 -- 172.21.15.12:0/1837097515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a41a5f90 con 0x7f83a40773e0 2024-03-17T23:54:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.613+0000 7f83a2000640 1 -- 172.21.15.12:0/1837097515 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f838c0035f0 con 0x7f83a40773e0 2024-03-17T23:54:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.613+0000 7f83a2000640 1 -- 172.21.15.12:0/1837097515 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c005220 con 0x7f83a40773e0 2024-03-17T23:54:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.613+0000 7f83ab800640 1 -- 172.21.15.12:0/1837097515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a41a64a0 con 0x7f83a40773e0 2024-03-17T23:54:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.614+0000 7f83ab800640 1 -- 172.21.15.12:0/1837097515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83a41104b0 con 0x7f83a40773e0 2024-03-17T23:54:21.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.616+0000 7f83a2000640 1 -- 172.21.15.12:0/1837097515 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f838c0037a0 con 0x7f83a40773e0 2024-03-17T23:54:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.617+0000 7f83a2000640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83880a2800 0x7f83880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.618+0000 7f83a2000640 1 -- 172.21.15.12:0/1837097515 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f838c0cebe0 con 0x7f83a40773e0 2024-03-17T23:54:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.618+0000 7f83a8e00640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83880a2800 0x7f83880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:21.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.618+0000 7f83a8e00640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83880a2800 0x7f83880a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8394005fd0 tx=0x7f8394009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.620+0000 7f83a2000640 1 -- 172.21.15.12:0/1837097515 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f838c09c080 con 0x7f83a40773e0 2024-03-17T23:54:21.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:21 smithi012 ceph-mon[31541]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:21.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:21 smithi012 ceph-mon[31541]: Deploying daemon mds.d on smithi012 2024-03-17T23:54:21.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:21 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.12:6834/1091924746,v1:172.21.15.12:6835/1091924746] up:boot 2024-03-17T23:54:21.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:21 smithi012 ceph-mon[31541]: fsmap 3 up:standby 2024-03-17T23:54:21.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:21 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-17T23:54:21.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:21 smithi079 ceph-mon[36241]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:21.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:21 smithi079 ceph-mon[36241]: Deploying daemon mds.d on smithi012 2024-03-17T23:54:21.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:21 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.12:6834/1091924746,v1:172.21.15.12:6835/1091924746] up:boot 2024-03-17T23:54:21.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:21 smithi079 ceph-mon[36241]: fsmap 3 up:standby 2024-03-17T23:54:21.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:21 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-17T23:54:21.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.871+0000 7f83ab800640 1 -- 172.21.15.12:0/1837097515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83a402d1b0 con 0x7f83a40773e0 2024-03-17T23:54:21.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.872+0000 7f83a2000640 1 -- 172.21.15.12:0/1837097515 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f838c09b980 con 0x7f83a40773e0 2024-03-17T23:54:21.873 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:21.873 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-17T23:54:21.873 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 4 2024-03-17T23:54:21.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.874+0000 7f83ab800640 1 -- 172.21.15.12:0/1837097515 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83880a2800 msgr2=0x7f83880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:21.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.874+0000 7f83ab800640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83880a2800 0x7f83880a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8394005fd0 tx=0x7f8394009500 comp rx=0 tx=0).stop 2024-03-17T23:54:21.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.874+0000 7f83ab800640 1 -- 172.21.15.12:0/1837097515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83a40773e0 msgr2=0x7f83a419e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:21.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.874+0000 7f83ab800640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83a40773e0 0x7f83a419e5c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f838c00e9f0 tx=0x7f838c00eeb0 comp rx=0 tx=0).stop 2024-03-17T23:54:21.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.875+0000 7f83ab800640 1 -- 172.21.15.12:0/1837097515 shutdown_connections 2024-03-17T23:54:21.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.875+0000 7f83ab800640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83880a2800 0x7f83880a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:21.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.875+0000 7f83ab800640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83a4077d80 0x7f83a41a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:21.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.875+0000 7f83ab800640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83a40773e0 0x7f83a419e5c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:21.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.875+0000 7f83ab800640 1 --2- 172.21.15.12:0/1837097515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f83a40761a0 0x7f83a419e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:21.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.875+0000 7f83ab800640 1 -- 172.21.15.12:0/1837097515 >> 172.21.15.12:0/1837097515 conn(0x7f83a40fef90 msgr2=0x7f83a41005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:21.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.875+0000 7f83ab800640 1 -- 172.21.15.12:0/1837097515 shutdown_connections 2024-03-17T23:54:21.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:21.875+0000 7f83ab800640 1 -- 172.21.15.12:0/1837097515 wait complete. 2024-03-17T23:54:22.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:22 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1837097515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:22.629 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:22 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1837097515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:22.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:22 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1837097515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:23.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:23 smithi012 ceph-mon[31541]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:23.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:23 smithi191 ceph-mon[36708]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:23.793 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 17 23:54:23 smithi012 systemd[1]: Starting Ceph mds.d for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-17T23:54:23.793 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 17 23:54:23 smithi012 podman[69437]: 2024-03-17 23:54:23.697257856 +0000 UTC m=+0.039329651 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-17T23:54:23.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:23 smithi079 ceph-mon[36241]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:23.896 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:24.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.124+0000 7f30c2400640 1 -- 172.21.15.12:0/452134719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30bc1033a0 msgr2=0x7f30bc109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:24.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.124+0000 7f30c2400640 1 --2- 172.21.15.12:0/452134719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30bc1033a0 0x7f30bc109c30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009ca0 tx=0x7f30b402c100 comp rx=0 tx=0).stop 2024-03-17T23:54:24.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.124+0000 7f30c2400640 1 -- 172.21.15.12:0/452134719 shutdown_connections 2024-03-17T23:54:24.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.124+0000 7f30c2400640 1 --2- 172.21.15.12:0/452134719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30bc1033a0 0x7f30bc109c30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:24.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.124+0000 7f30c2400640 1 --2- 172.21.15.12:0/452134719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30bc1029e0 0x7f30bc102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:24.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.124+0000 7f30c2400640 1 --2- 172.21.15.12:0/452134719 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30bc1017f0 0x7f30bc101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:24.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.124+0000 7f30c2400640 1 -- 172.21.15.12:0/452134719 >> 172.21.15.12:0/452134719 conn(0x7f30bc0fcd80 msgr2=0x7f30bc0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:24.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.124+0000 7f30c2400640 1 -- 172.21.15.12:0/452134719 shutdown_connections 2024-03-17T23:54:24.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.124+0000 7f30c2400640 1 -- 172.21.15.12:0/452134719 wait complete. 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.125+0000 7f30c2400640 1 Processor -- start 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.125+0000 7f30c2400640 1 -- start start 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.125+0000 7f30c2400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30bc1017f0 0x7f30bc075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.125+0000 7f30c2400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30bc1029e0 0x7f30bc073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.125+0000 7f30c2400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30bc1033a0 0x7f30bc074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.125+0000 7f30c2400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc10d450 con 0x7f30bc1033a0 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.125+0000 7f30c2400640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc10d2d0 con 0x7f30bc1017f0 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.125+0000 7f30c2400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc10d5d0 con 0x7f30bc1029e0 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.126+0000 7f30bbe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30bc1033a0 0x7f30bc074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.126+0000 7f30bb400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30bc1017f0 0x7f30bc075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.126+0000 7f30baa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30bc1029e0 0x7f30bc073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.126+0000 7f30bb400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30bc1017f0 0x7f30bc075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56170/0 (socket says 172.21.15.12:56170) 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.126+0000 7f30baa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30bc1029e0 0x7f30bc073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:47530/0 (socket says 172.21.15.12:47530) 2024-03-17T23:54:24.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.126+0000 7f30bb400640 1 -- 172.21.15.12:0/2679048572 learned_addr learned my addr 172.21.15.12:0/2679048572 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:24.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.126+0000 7f30baa00640 1 -- 172.21.15.12:0/2679048572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30bc1017f0 msgr2=0x7f30bc075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:24.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.126+0000 7f30baa00640 1 --2- 172.21.15.12:0/2679048572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30bc1017f0 0x7f30bc075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:24.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.126+0000 7f30baa00640 1 -- 172.21.15.12:0/2679048572 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30bc1033a0 msgr2=0x7f30bc074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:24.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.126+0000 7f30baa00640 1 --2- 172.21.15.12:0/2679048572 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30bc1033a0 0x7f30bc074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:24.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.126+0000 7f30baa00640 1 -- 172.21.15.12:0/2679048572 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30bc0749a0 con 0x7f30bc1029e0 2024-03-17T23:54:24.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.126+0000 7f30bb400640 1 --2- 172.21.15.12:0/2679048572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30bc1017f0 0x7f30bc075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:24.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.127+0000 7f30baa00640 1 --2- 172.21.15.12:0/2679048572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30bc1029e0 0x7f30bc073bd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f30a400e9f0 tx=0x7f30a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:24.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.127+0000 7f30b3e00640 1 -- 172.21.15.12:0/2679048572 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a400ccc0 con 0x7f30bc1029e0 2024-03-17T23:54:24.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.127+0000 7f30c2400640 1 -- 172.21.15.12:0/2679048572 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30bc1a1b80 con 0x7f30bc1029e0 2024-03-17T23:54:24.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.127+0000 7f30b3e00640 1 -- 172.21.15.12:0/2679048572 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30a4010910 con 0x7f30bc1029e0 2024-03-17T23:54:24.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.127+0000 7f30b3e00640 1 -- 172.21.15.12:0/2679048572 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a4018960 con 0x7f30bc1029e0 2024-03-17T23:54:24.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.127+0000 7f30c2400640 1 -- 172.21.15.12:0/2679048572 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30bc1a2120 con 0x7f30bc1029e0 2024-03-17T23:54:24.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.128+0000 7f30c2400640 1 -- 172.21.15.12:0/2679048572 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30bc108b10 con 0x7f30bc1029e0 2024-03-17T23:54:24.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.130+0000 7f30b3e00640 1 -- 172.21.15.12:0/2679048572 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f30a4010ac0 con 0x7f30bc1029e0 2024-03-17T23:54:24.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.132+0000 7f30b3e00640 1 --2- 172.21.15.12:0/2679048572 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f30a00a2800 0x7f30a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:24.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.132+0000 7f30b3e00640 1 -- 172.21.15.12:0/2679048572 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f30a40ce9d0 con 0x7f30bc1029e0 2024-03-17T23:54:24.133 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 17 23:54:23 smithi012 podman[69437]: 2024-03-17 23:54:23.822738022 +0000 UTC m=+0.164809835 container create d0bed2ffb65e28296c3a377a34f5f67a0fc9cca2f710776df4f223c4ef376442 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-17T23:54:24.133 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 17 23:54:24 smithi012 podman[69437]: 2024-03-17 23:54:24.133063436 +0000 UTC m=+0.475135208 container init d0bed2ffb65e28296c3a377a34f5f67a0fc9cca2f710776df4f223c4ef376442 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-17T23:54:24.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.132+0000 7f30bb400640 1 --2- 172.21.15.12:0/2679048572 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f30a00a2800 0x7f30a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:24.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.133+0000 7f30bb400640 1 --2- 172.21.15.12:0/2679048572 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f30a00a2800 0x7f30a00a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0058c0 tx=0x7f30ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:24.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.134+0000 7f30b3e00640 1 -- 172.21.15.12:0/2679048572 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30a409b820 con 0x7f30bc1029e0 2024-03-17T23:54:24.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.393+0000 7f30c2400640 1 -- 172.21.15.12:0/2679048572 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30bc02d1b0 con 0x7f30bc1029e0 2024-03-17T23:54:24.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.394+0000 7f30b3e00640 1 -- 172.21.15.12:0/2679048572 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f30a409f8c0 con 0x7f30bc1029e0 2024-03-17T23:54:24.395 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:24.395 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-03-17T23:54:24.395 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 5 2024-03-17T23:54:24.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.397+0000 7f30c2400640 1 -- 172.21.15.12:0/2679048572 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f30a00a2800 msgr2=0x7f30a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:24.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.397+0000 7f30c2400640 1 --2- 172.21.15.12:0/2679048572 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f30a00a2800 0x7f30a00a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0058c0 tx=0x7f30ac005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:24.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.397+0000 7f30c2400640 1 -- 172.21.15.12:0/2679048572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30bc1029e0 msgr2=0x7f30bc073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:24.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.397+0000 7f30c2400640 1 --2- 172.21.15.12:0/2679048572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30bc1029e0 0x7f30bc073bd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f30a400e9f0 tx=0x7f30a400eeb0 comp rx=0 tx=0).stop 2024-03-17T23:54:24.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.397+0000 7f30c2400640 1 -- 172.21.15.12:0/2679048572 shutdown_connections 2024-03-17T23:54:24.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.397+0000 7f30c2400640 1 --2- 172.21.15.12:0/2679048572 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f30a00a2800 0x7f30a00a4cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:24.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.397+0000 7f30c2400640 1 --2- 172.21.15.12:0/2679048572 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30bc1033a0 0x7f30bc074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:24.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.397+0000 7f30c2400640 1 --2- 172.21.15.12:0/2679048572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30bc1029e0 0x7f30bc073bd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:24.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.397+0000 7f30c2400640 1 --2- 172.21.15.12:0/2679048572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30bc1017f0 0x7f30bc075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:24.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.397+0000 7f30c2400640 1 -- 172.21.15.12:0/2679048572 >> 172.21.15.12:0/2679048572 conn(0x7f30bc0fcd80 msgr2=0x7f30bc105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:24.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.397+0000 7f30c2400640 1 -- 172.21.15.12:0/2679048572 shutdown_connections 2024-03-17T23:54:24.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:24.397+0000 7f30c2400640 1 -- 172.21.15.12:0/2679048572 wait complete. 2024-03-17T23:54:24.413 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 17 23:54:24 smithi012 podman[69437]: 2024-03-17 23:54:24.140680361 +0000 UTC m=+0.482752141 container start d0bed2ffb65e28296c3a377a34f5f67a0fc9cca2f710776df4f223c4ef376442 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-17T23:54:24.413 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 17 23:54:24 smithi012 ceph-mds[69495]: -- 172.21.15.12:0/3240393446 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5631619a5a40 con 0x563161978c00 2024-03-17T23:54:24.413 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 17 23:54:24 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d[69469]: starting mds.d at 2024-03-17T23:54:24.414 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 17 23:54:24 smithi012 bash[69437]: d0bed2ffb65e28296c3a377a34f5f67a0fc9cca2f710776df4f223c4ef376442 2024-03-17T23:54:24.414 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 17 23:54:24 smithi012 systemd[1]: Started Ceph mds.d for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:54:25.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:25 smithi191 ceph-mon[36708]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:25.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:25 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.12:6836/1104379442,v1:172.21.15.12:6837/1104379442] up:boot 2024-03-17T23:54:25.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:25 smithi191 ceph-mon[36708]: fsmap 4 up:standby 2024-03-17T23:54:25.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-17T23:54:25.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:25.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:25.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:25 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2679048572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:25.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:25.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:25.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:25.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:25 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:25.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:25 smithi191 ceph-mon[36708]: Deploying daemon mds.e on smithi079 2024-03-17T23:54:25.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:25 smithi012 ceph-mon[31541]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:25.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:25 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.12:6836/1104379442,v1:172.21.15.12:6837/1104379442] up:boot 2024-03-17T23:54:25.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:25 smithi012 ceph-mon[31541]: fsmap 4 up:standby 2024-03-17T23:54:25.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-17T23:54:25.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:25.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:25.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:25 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2679048572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:25.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:25.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:25.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:25.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:25 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:25.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:25 smithi012 ceph-mon[31541]: Deploying daemon mds.e on smithi079 2024-03-17T23:54:25.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:25 smithi079 ceph-mon[36241]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:25.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:25 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.12:6836/1104379442,v1:172.21.15.12:6837/1104379442] up:boot 2024-03-17T23:54:25.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:25 smithi079 ceph-mon[36241]: fsmap 4 up:standby 2024-03-17T23:54:25.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-17T23:54:25.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:25.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:25.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:25 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2679048572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:25.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:25.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:25.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:25.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:25 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:25.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:25 smithi079 ceph-mon[36241]: Deploying daemon mds.e on smithi079 2024-03-17T23:54:26.417 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.637+0000 7f45a6400640 1 -- 172.21.15.12:0/2725990814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0077d80 msgr2=0x7f45a0078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.637+0000 7f45a6400640 1 --2- 172.21.15.12:0/2725990814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0077d80 0x7f45a0078620 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4598009ca0 tx=0x7f459802c0f0 comp rx=0 tx=0).stop 2024-03-17T23:54:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.637+0000 7f45a6400640 1 -- 172.21.15.12:0/2725990814 shutdown_connections 2024-03-17T23:54:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.637+0000 7f45a6400640 1 --2- 172.21.15.12:0/2725990814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0077d80 0x7f45a0078620 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.637+0000 7f45a6400640 1 --2- 172.21.15.12:0/2725990814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45a00773e0 0x7f45a0077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.637+0000 7f45a6400640 1 --2- 172.21.15.12:0/2725990814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45a00761a0 0x7f45a00765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.637+0000 7f45a6400640 1 -- 172.21.15.12:0/2725990814 >> 172.21.15.12:0/2725990814 conn(0x7f45a00fed70 msgr2=0x7f45a01011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.637+0000 7f45a6400640 1 -- 172.21.15.12:0/2725990814 shutdown_connections 2024-03-17T23:54:26.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.637+0000 7f45a6400640 1 -- 172.21.15.12:0/2725990814 wait complete. 2024-03-17T23:54:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.638+0000 7f45a6400640 1 Processor -- start 2024-03-17T23:54:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.638+0000 7f45a6400640 1 -- start start 2024-03-17T23:54:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a6400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45a00761a0 0x7f45a019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a6400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45a00773e0 0x7f45a019e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a6400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0077d80 0x7f45a01a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a6400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0113890 con 0x7f45a0077d80 2024-03-17T23:54:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a6400640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0113710 con 0x7f45a00773e0 2024-03-17T23:54:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a6400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0113a10 con 0x7f45a00761a0 2024-03-17T23:54:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a5a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0077d80 0x7f45a01a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f459fe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45a00773e0 0x7f45a019e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a5a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0077d80 0x7f45a01a5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53172/0 (socket says 172.21.15.12:53172) 2024-03-17T23:54:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a5a00640 1 -- 172.21.15.12:0/1625897833 learned_addr learned my addr 172.21.15.12:0/1625897833 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a5000640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45a00761a0 0x7f45a019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a5a00640 1 -- 172.21.15.12:0/1625897833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45a00761a0 msgr2=0x7f45a019de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a5a00640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45a00761a0 0x7f45a019de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a5a00640 1 -- 172.21.15.12:0/1625897833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45a00773e0 msgr2=0x7f45a019e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a5a00640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45a00773e0 0x7f45a019e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f45a5a00640 1 -- 172.21.15.12:0/1625897833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a01a5ab0 con 0x7f45a0077d80 2024-03-17T23:54:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.639+0000 7f459fe00640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45a00773e0 0x7f45a019e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.640+0000 7f45a5a00640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0077d80 0x7f45a01a5440 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4598037040 tx=0x7f4598032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.640+0000 7f459d600640 1 -- 172.21.15.12:0/1625897833 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598041020 con 0x7f45a0077d80 2024-03-17T23:54:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.640+0000 7f459d600640 1 -- 172.21.15.12:0/1625897833 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4598034cd0 con 0x7f45a0077d80 2024-03-17T23:54:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.640+0000 7f459d600640 1 -- 172.21.15.12:0/1625897833 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598039040 con 0x7f45a0077d80 2024-03-17T23:54:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.640+0000 7f45a6400640 1 -- 172.21.15.12:0/1625897833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a01a5d40 con 0x7f45a0077d80 2024-03-17T23:54:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.640+0000 7f45a6400640 1 -- 172.21.15.12:0/1625897833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a01a6180 con 0x7f45a0077d80 2024-03-17T23:54:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.641+0000 7f45a6400640 1 -- 172.21.15.12:0/1625897833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45a0110170 con 0x7f45a0077d80 2024-03-17T23:54:26.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.645+0000 7f459d600640 1 -- 172.21.15.12:0/1625897833 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4598044020 con 0x7f45a0077d80 2024-03-17T23:54:26.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.646+0000 7f459d600640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f45880a2800 0x7f45880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.646+0000 7f459d600640 1 -- 172.21.15.12:0/1625897833 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f45980f0940 con 0x7f45a0077d80 2024-03-17T23:54:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.646+0000 7f45a5000640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f45880a2800 0x7f45880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.647+0000 7f45a5000640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f45880a2800 0x7f45880a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f45940095f0 tx=0x7f4594005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:26.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.649+0000 7f459d600640 1 -- 172.21.15.12:0/1625897833 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f459804b020 con 0x7f45a0077d80 2024-03-17T23:54:26.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.892+0000 7f45a6400640 1 -- 172.21.15.12:0/1625897833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45a002d250 con 0x7f45a0077d80 2024-03-17T23:54:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.893+0000 7f459d600640 1 -- 172.21.15.12:0/1625897833 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f45980bde70 con 0x7f45a0077d80 2024-03-17T23:54:26.894 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:26.894 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-03-17T23:54:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 5 2024-03-17T23:54:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.896+0000 7f45a6400640 1 -- 172.21.15.12:0/1625897833 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f45880a2800 msgr2=0x7f45880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.896+0000 7f45a6400640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f45880a2800 0x7f45880a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f45940095f0 tx=0x7f4594005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.896+0000 7f45a6400640 1 -- 172.21.15.12:0/1625897833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0077d80 msgr2=0x7f45a01a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.896+0000 7f45a6400640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0077d80 0x7f45a01a5440 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4598037040 tx=0x7f4598032f40 comp rx=0 tx=0).stop 2024-03-17T23:54:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.896+0000 7f45a6400640 1 -- 172.21.15.12:0/1625897833 shutdown_connections 2024-03-17T23:54:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.896+0000 7f45a6400640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f45880a2800 0x7f45880a4cc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.896+0000 7f45a6400640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0077d80 0x7f45a01a5440 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.896+0000 7f45a6400640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45a00773e0 0x7f45a019e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.896+0000 7f45a6400640 1 --2- 172.21.15.12:0/1625897833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45a00761a0 0x7f45a019de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.896+0000 7f45a6400640 1 -- 172.21.15.12:0/1625897833 >> 172.21.15.12:0/1625897833 conn(0x7f45a00fed70 msgr2=0x7f45a0100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.896+0000 7f45a6400640 1 -- 172.21.15.12:0/1625897833 shutdown_connections 2024-03-17T23:54:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:26.897+0000 7f45a6400640 1 -- 172.21.15.12:0/1625897833 wait complete. 2024-03-17T23:54:27.046 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 17 23:54:26 smithi079 podman[65639]: 2024-03-17 23:54:26.938059559 +0000 UTC m=+0.142265465 container create c3b0811d0d33bceb4607f580ba60889522d087bbe244500c3f0b48cb39a73f2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-17T23:54:27.332 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 17 23:54:27 smithi079 podman[65639]: 2024-03-17 23:54:27.238458397 +0000 UTC m=+0.442664310 container init c3b0811d0d33bceb4607f580ba60889522d087bbe244500c3f0b48cb39a73f2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-17T23:54:27.333 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 17 23:54:27 smithi079 podman[65639]: 2024-03-17 23:54:27.241201417 +0000 UTC m=+0.445407334 container start c3b0811d0d33bceb4607f580ba60889522d087bbe244500c3f0b48cb39a73f2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-17T23:54:27.333 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 17 23:54:27 smithi079 ceph-mds[65658]: -- 172.21.15.79:0/657598450 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f53030ba40 con 0x55f5302df000 2024-03-17T23:54:27.333 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 17 23:54:27 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-e[65654]: starting mds.e at 2024-03-17T23:54:27.333 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 17 23:54:27 smithi079 bash[65639]: c3b0811d0d33bceb4607f580ba60889522d087bbe244500c3f0b48cb39a73f2a 2024-03-17T23:54:27.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:27 smithi012 ceph-mon[31541]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:27.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:27 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1625897833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:27.795 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 17 23:54:27 smithi079 systemd[1]: Started Ceph mds.e for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:54:27.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:27 smithi079 ceph-mon[36241]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:27.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:27 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1625897833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:27.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:27 smithi191 ceph-mon[36708]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:27.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:27 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1625897833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:28.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:28 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.79:6834/4099928196,v1:172.21.15.79:6835/4099928196] up:boot 2024-03-17T23:54:28.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:28 smithi191 ceph-mon[36708]: fsmap 5 up:standby 2024-03-17T23:54:28.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-17T23:54:28.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:28.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:28.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:28.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:28.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:28.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:28.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:28 smithi191 ceph-mon[36708]: Deploying daemon mds.f on smithi191 2024-03-17T23:54:28.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:28 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.79:6834/4099928196,v1:172.21.15.79:6835/4099928196] up:boot 2024-03-17T23:54:28.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:28 smithi012 ceph-mon[31541]: fsmap 5 up:standby 2024-03-17T23:54:28.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-17T23:54:28.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:28.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:28.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:28.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:28.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:28.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:28.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:28 smithi012 ceph-mon[31541]: Deploying daemon mds.f on smithi191 2024-03-17T23:54:28.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:28 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.79:6834/4099928196,v1:172.21.15.79:6835/4099928196] up:boot 2024-03-17T23:54:28.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:28 smithi079 ceph-mon[36241]: fsmap 5 up:standby 2024-03-17T23:54:28.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-17T23:54:28.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:28.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:28.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:28.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:28.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:28.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:28.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:28 smithi079 ceph-mon[36241]: Deploying daemon mds.f on smithi191 2024-03-17T23:54:28.920 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.148+0000 7f5ac9600640 1 -- 172.21.15.12:0/2030319543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ac4103810 msgr2=0x7f5ac4103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.148+0000 7f5ac9600640 1 --2- 172.21.15.12:0/2030319543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ac4103810 0x7f5ac4103c30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009ca0 tx=0x7f5abc02c0f0 comp rx=0 tx=0).stop 2024-03-17T23:54:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.148+0000 7f5ac9600640 1 -- 172.21.15.12:0/2030319543 shutdown_connections 2024-03-17T23:54:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.148+0000 7f5ac9600640 1 --2- 172.21.15.12:0/2030319543 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ac41053c0 0x7f5ac410bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.148+0000 7f5ac9600640 1 --2- 172.21.15.12:0/2030319543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac4104a00 0x7f5ac4104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.148+0000 7f5ac9600640 1 --2- 172.21.15.12:0/2030319543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ac4103810 0x7f5ac4103c30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.148+0000 7f5ac9600640 1 -- 172.21.15.12:0/2030319543 >> 172.21.15.12:0/2030319543 conn(0x7f5ac40fed80 msgr2=0x7f5ac41011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.149+0000 7f5ac9600640 1 -- 172.21.15.12:0/2030319543 shutdown_connections 2024-03-17T23:54:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.149+0000 7f5ac9600640 1 -- 172.21.15.12:0/2030319543 wait complete. 2024-03-17T23:54:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.149+0000 7f5ac9600640 1 Processor -- start 2024-03-17T23:54:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.150+0000 7f5ac9600640 1 -- start start 2024-03-17T23:54:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.150+0000 7f5ac9600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ac4103810 0x7f5ac4199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.150+0000 7f5ac9600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ac4104a00 0x7f5ac419a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.150+0000 7f5ac9600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac41053c0 0x7f5ac41a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.150+0000 7f5ac9600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac410f490 con 0x7f5ac41053c0 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.150+0000 7f5ac9600640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac410f310 con 0x7f5ac4103810 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.150+0000 7f5ac9600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac410f610 con 0x7f5ac4104a00 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.150+0000 7f5ac8c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac41053c0 0x7f5ac41a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.150+0000 7f5ac3400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ac4104a00 0x7f5ac419a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.150+0000 7f5ac3e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ac4103810 0x7f5ac4199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.150+0000 7f5ac8c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac41053c0 0x7f5ac41a1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53178/0 (socket says 172.21.15.12:53178) 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.150+0000 7f5ac8c00640 1 -- 172.21.15.12:0/291475970 learned_addr learned my addr 172.21.15.12:0/291475970 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.151+0000 7f5ac3400640 1 -- 172.21.15.12:0/291475970 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ac4103810 msgr2=0x7f5ac4199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.151+0000 7f5ac3400640 1 --2- 172.21.15.12:0/291475970 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ac4103810 0x7f5ac4199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.151+0000 7f5ac3400640 1 -- 172.21.15.12:0/291475970 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac41053c0 msgr2=0x7f5ac41a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.151+0000 7f5ac3400640 1 --2- 172.21.15.12:0/291475970 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac41053c0 0x7f5ac41a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.151+0000 7f5ac3400640 1 -- 172.21.15.12:0/291475970 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac41a1700 con 0x7f5ac4104a00 2024-03-17T23:54:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.151+0000 7f5ac3400640 1 --2- 172.21.15.12:0/291475970 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ac4104a00 0x7f5ac419a020 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000e9f0 tx=0x7f5ab000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.151+0000 7f5ac0c00640 1 -- 172.21.15.12:0/291475970 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab000ccc0 con 0x7f5ac4104a00 2024-03-17T23:54:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.151+0000 7f5ac9600640 1 -- 172.21.15.12:0/291475970 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac41a19f0 con 0x7f5ac4104a00 2024-03-17T23:54:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.151+0000 7f5ac0c00640 1 -- 172.21.15.12:0/291475970 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ab0010910 con 0x7f5ac4104a00 2024-03-17T23:54:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.151+0000 7f5ac0c00640 1 -- 172.21.15.12:0/291475970 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0018960 con 0x7f5ac4104a00 2024-03-17T23:54:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.152+0000 7f5ac9600640 1 -- 172.21.15.12:0/291475970 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac41a1f90 con 0x7f5ac4104a00 2024-03-17T23:54:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.153+0000 7f5ac9600640 1 -- 172.21.15.12:0/291475970 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac410aad0 con 0x7f5ac4104a00 2024-03-17T23:54:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.155+0000 7f5ac0c00640 1 -- 172.21.15.12:0/291475970 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5ab0010ac0 con 0x7f5ac4104a00 2024-03-17T23:54:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.157+0000 7f5ac0c00640 1 --2- 172.21.15.12:0/291475970 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5aa80a2800 0x7f5aa80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.157+0000 7f5ac0c00640 1 -- 172.21.15.12:0/291475970 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5ab00ce9d0 con 0x7f5ac4104a00 2024-03-17T23:54:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.157+0000 7f5ac3e00640 1 --2- 172.21.15.12:0/291475970 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5aa80a2800 0x7f5aa80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.158+0000 7f5ac3e00640 1 --2- 172.21.15.12:0/291475970 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5aa80a2800 0x7f5aa80a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5abc02c630 tx=0x7f5abc009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.158+0000 7f5ac0c00640 1 -- 172.21.15.12:0/291475970 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5ab009b820 con 0x7f5ac4104a00 2024-03-17T23:54:29.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.401+0000 7f5ac9600640 1 -- 172.21.15.12:0/291475970 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ac402d370 con 0x7f5ac4104a00 2024-03-17T23:54:29.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.402+0000 7f5ac0c00640 1 -- 172.21.15.12:0/291475970 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7f5ab009f8c0 con 0x7f5ac4104a00 2024-03-17T23:54:29.403 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:29.403 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-17T23:54:29.403 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 6 2024-03-17T23:54:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.405+0000 7f5ac9600640 1 -- 172.21.15.12:0/291475970 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5aa80a2800 msgr2=0x7f5aa80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.405+0000 7f5ac9600640 1 --2- 172.21.15.12:0/291475970 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5aa80a2800 0x7f5aa80a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5abc02c630 tx=0x7f5abc009360 comp rx=0 tx=0).stop 2024-03-17T23:54:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.405+0000 7f5ac9600640 1 -- 172.21.15.12:0/291475970 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ac4104a00 msgr2=0x7f5ac419a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.405+0000 7f5ac9600640 1 --2- 172.21.15.12:0/291475970 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ac4104a00 0x7f5ac419a020 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000e9f0 tx=0x7f5ab000eeb0 comp rx=0 tx=0).stop 2024-03-17T23:54:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.405+0000 7f5ac9600640 1 -- 172.21.15.12:0/291475970 shutdown_connections 2024-03-17T23:54:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.405+0000 7f5ac9600640 1 --2- 172.21.15.12:0/291475970 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5aa80a2800 0x7f5aa80a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.405+0000 7f5ac9600640 1 --2- 172.21.15.12:0/291475970 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ac41053c0 0x7f5ac41a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.405+0000 7f5ac9600640 1 --2- 172.21.15.12:0/291475970 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ac4104a00 0x7f5ac419a020 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.405+0000 7f5ac9600640 1 --2- 172.21.15.12:0/291475970 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ac4103810 0x7f5ac4199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.405+0000 7f5ac9600640 1 -- 172.21.15.12:0/291475970 >> 172.21.15.12:0/291475970 conn(0x7f5ac40fed80 msgr2=0x7f5ac4107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.405+0000 7f5ac9600640 1 -- 172.21.15.12:0/291475970 shutdown_connections 2024-03-17T23:54:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:29.405+0000 7f5ac9600640 1 -- 172.21.15.12:0/291475970 wait complete. 2024-03-17T23:54:29.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:29 smithi191 ceph-mon[36708]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:29.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:29 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/291475970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:29.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:29 smithi079 ceph-mon[36241]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:29.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:29 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/291475970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:29.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:29 smithi012 ceph-mon[31541]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:29.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:29 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/291475970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:29.986 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 17 23:54:29 smithi191 systemd[1]: Starting Ceph mds.f for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-17T23:54:30.258 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 17 23:54:29 smithi191 podman[67182]: 2024-03-17 23:54:29.889302678 +0000 UTC m=+0.018823618 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-17T23:54:30.258 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 17 23:54:30 smithi191 podman[67182]: 2024-03-17 23:54:30.021797261 +0000 UTC m=+0.151318180 container create fad973357eed9716973bcce61deb5bf2e171ffa2c85c2fdccdc99e031d31ba68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-f, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-17T23:54:30.258 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 17 23:54:30 smithi191 podman[67182]: 2024-03-17 23:54:30.258631469 +0000 UTC m=+0.388152414 container init fad973357eed9716973bcce61deb5bf2e171ffa2c85c2fdccdc99e031d31ba68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-f, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-17T23:54:30.591 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 17 23:54:30 smithi191 podman[67182]: 2024-03-17 23:54:30.26142905 +0000 UTC m=+0.390949978 container start fad973357eed9716973bcce61deb5bf2e171ffa2c85c2fdccdc99e031d31ba68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-17T23:54:30.591 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 17 23:54:30 smithi191 ceph-mds[67201]: -- 172.21.15.191:0/2679244031 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5566c2dc3a40 con 0x5566c2d96c00 2024-03-17T23:54:30.591 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 17 23:54:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-f[67197]: starting mds.f at 2024-03-17T23:54:30.591 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 17 23:54:30 smithi191 bash[67182]: fad973357eed9716973bcce61deb5bf2e171ffa2c85c2fdccdc99e031d31ba68 2024-03-17T23:54:30.591 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 17 23:54:30 smithi191 systemd[1]: Started Ceph mds.f for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:54:31.426 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.659+0000 7fefe4e00640 1 -- 172.21.15.12:0/1626737506 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe01097d0 msgr2=0x7fefe0110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.659+0000 7fefe4e00640 1 --2- 172.21.15.12:0/1626737506 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe01097d0 0x7fefe0110060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fefd8009ca0 tx=0x7fefd802c0f0 comp rx=0 tx=0).stop 2024-03-17T23:54:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.660+0000 7fefe4e00640 1 -- 172.21.15.12:0/1626737506 shutdown_connections 2024-03-17T23:54:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.660+0000 7fefe4e00640 1 --2- 172.21.15.12:0/1626737506 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe01097d0 0x7fefe0110060 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.660+0000 7fefe4e00640 1 --2- 172.21.15.12:0/1626737506 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefe0108e10 0x7fefe0109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.660+0000 7fefe4e00640 1 --2- 172.21.15.12:0/1626737506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefe0107c20 0x7fefe0108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.660+0000 7fefe4e00640 1 -- 172.21.15.12:0/1626737506 >> 172.21.15.12:0/1626737506 conn(0x7fefe0076bb0 msgr2=0x7fefe0078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.660+0000 7fefe4e00640 1 -- 172.21.15.12:0/1626737506 shutdown_connections 2024-03-17T23:54:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.660+0000 7fefe4e00640 1 -- 172.21.15.12:0/1626737506 wait complete. 2024-03-17T23:54:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.661+0000 7fefe4e00640 1 Processor -- start 2024-03-17T23:54:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.661+0000 7fefe4e00640 1 -- start start 2024-03-17T23:54:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.661+0000 7fefe4e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefe0107c20 0x7fefe019dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefe4e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefe0108e10 0x7fefe019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefe4e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe01097d0 0x7fefe01a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefe4e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fefe0113880 con 0x7fefe01097d0 2024-03-17T23:54:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefe4e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fefe0113700 con 0x7fefe0107c20 2024-03-17T23:54:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefe4e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fefe0113a00 con 0x7fefe0108e10 2024-03-17T23:54:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefdfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe01097d0 0x7fefe01a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefdf400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefe0107c20 0x7fefe019dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefdf400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefe0107c20 0x7fefe019dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56230/0 (socket says 172.21.15.12:56230) 2024-03-17T23:54:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefdf400640 1 -- 172.21.15.12:0/3274924614 learned_addr learned my addr 172.21.15.12:0/3274924614 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefdfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe01097d0 0x7fefe01a5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53200/0 (socket says 172.21.15.12:53200) 2024-03-17T23:54:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefd7e00640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefe0108e10 0x7fefe019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefdfe00640 1 -- 172.21.15.12:0/3274924614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefe0108e10 msgr2=0x7fefe019e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefdfe00640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefe0108e10 0x7fefe019e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefdfe00640 1 -- 172.21.15.12:0/3274924614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefe0107c20 msgr2=0x7fefe019dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefdfe00640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefe0107c20 0x7fefe019dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefdfe00640 1 -- 172.21.15.12:0/3274924614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe01a5b00 con 0x7fefe01097d0 2024-03-17T23:54:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefd7e00640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefe0108e10 0x7fefe019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.662+0000 7fefdf400640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefe0107c20 0x7fefe019dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.663+0000 7fefdfe00640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe01097d0 0x7fefe01a5490 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fefd802c700 tx=0x7fefd8032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.663+0000 7fefdcc00640 1 -- 172.21.15.12:0/3274924614 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd8041020 con 0x7fefe01097d0 2024-03-17T23:54:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.663+0000 7fefdcc00640 1 -- 172.21.15.12:0/3274924614 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefd8034cd0 con 0x7fefe01097d0 2024-03-17T23:54:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.663+0000 7fefdcc00640 1 -- 172.21.15.12:0/3274924614 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd8039040 con 0x7fefe01097d0 2024-03-17T23:54:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.663+0000 7fefe4e00640 1 -- 172.21.15.12:0/3274924614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe01a5d90 con 0x7fefe01097d0 2024-03-17T23:54:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.663+0000 7fefe4e00640 1 -- 172.21.15.12:0/3274924614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe01a6270 con 0x7fefe01097d0 2024-03-17T23:54:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.664+0000 7fefe4e00640 1 -- 172.21.15.12:0/3274924614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe010eee0 con 0x7fefe01097d0 2024-03-17T23:54:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.666+0000 7fefdcc00640 1 -- 172.21.15.12:0/3274924614 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fefd8044020 con 0x7fefe01097d0 2024-03-17T23:54:31.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:31 smithi012 ceph-mon[31541]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:31.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:31.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:31.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:31 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.191:6834/3192827956,v1:172.21.15.191:6835/3192827956] up:boot 2024-03-17T23:54:31.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:31 smithi012 ceph-mon[31541]: fsmap 6 up:standby 2024-03-17T23:54:31.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-17T23:54:31.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:31.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:31.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:31.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:31 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:31.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:31 smithi012 ceph-mon[31541]: Deploying daemon mds.c on smithi191 2024-03-17T23:54:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.668+0000 7fefdcc00640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fefc40a2880 0x7fefc40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.668+0000 7fefdcc00640 1 -- 172.21.15.12:0/3274924614 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fefd80f0730 con 0x7fefe01097d0 2024-03-17T23:54:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.668+0000 7fefdf400640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fefc40a2880 0x7fefc40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.669+0000 7fefdf400640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fefc40a2880 0x7fefc40a4d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fefd00058c0 tx=0x7fefd0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.671+0000 7fefdcc00640 1 -- 172.21.15.12:0/3274924614 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fefd804b020 con 0x7fefe01097d0 2024-03-17T23:54:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:31 smithi079 ceph-mon[36241]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:31 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.191:6834/3192827956,v1:172.21.15.191:6835/3192827956] up:boot 2024-03-17T23:54:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:31 smithi079 ceph-mon[36241]: fsmap 6 up:standby 2024-03-17T23:54:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-17T23:54:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:31 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:31 smithi079 ceph-mon[36241]: Deploying daemon mds.c on smithi191 2024-03-17T23:54:31.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:31 smithi191 ceph-mon[36708]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:31.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:31 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.191:6834/3192827956,v1:172.21.15.191:6835/3192827956] up:boot 2024-03-17T23:54:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:31 smithi191 ceph-mon[36708]: fsmap 6 up:standby 2024-03-17T23:54:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-17T23:54:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:31 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:31.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:31 smithi191 ceph-mon[36708]: Deploying daemon mds.c on smithi191 2024-03-17T23:54:31.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.917+0000 7fefe4e00640 1 -- 172.21.15.12:0/3274924614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefe002d1b0 con 0x7fefe01097d0 2024-03-17T23:54:31.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.918+0000 7fefdcc00640 1 -- 172.21.15.12:0/3274924614 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7fefd80bdc60 con 0x7fefe01097d0 2024-03-17T23:54:31.919 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:31.919 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-17T23:54:31.920 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-03-17T23:54:31.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.922+0000 7fefe4e00640 1 -- 172.21.15.12:0/3274924614 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fefc40a2880 msgr2=0x7fefc40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.922+0000 7fefe4e00640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fefc40a2880 0x7fefc40a4d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fefd00058c0 tx=0x7fefd0005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.922+0000 7fefe4e00640 1 -- 172.21.15.12:0/3274924614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe01097d0 msgr2=0x7fefe01a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.922+0000 7fefe4e00640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe01097d0 0x7fefe01a5490 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fefd802c700 tx=0x7fefd8032f20 comp rx=0 tx=0).stop 2024-03-17T23:54:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.922+0000 7fefe4e00640 1 -- 172.21.15.12:0/3274924614 shutdown_connections 2024-03-17T23:54:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.922+0000 7fefe4e00640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fefc40a2880 0x7fefc40a4d40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.922+0000 7fefe4e00640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefe01097d0 0x7fefe01a5490 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.922+0000 7fefe4e00640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefe0108e10 0x7fefe019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.922+0000 7fefe4e00640 1 --2- 172.21.15.12:0/3274924614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefe0107c20 0x7fefe019dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.922+0000 7fefe4e00640 1 -- 172.21.15.12:0/3274924614 >> 172.21.15.12:0/3274924614 conn(0x7fefe0076bb0 msgr2=0x7fefe010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.922+0000 7fefe4e00640 1 -- 172.21.15.12:0/3274924614 shutdown_connections 2024-03-17T23:54:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:31.922+0000 7fefe4e00640 1 -- 172.21.15.12:0/3274924614 wait complete. 2024-03-17T23:54:32.735 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:32 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3274924614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:32.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:32 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3274924614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:32.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:32 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3274924614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:33.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:33 smithi191 ceph-mon[36708]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:33.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:33.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:33 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:33.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:33 smithi079 ceph-mon[36241]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:33.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:33.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:33 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:33.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:33 smithi012 ceph-mon[31541]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:33.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:33.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:33 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:33.943 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:34.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.171+0000 7f0949000640 1 -- 172.21.15.12:0/4206290989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0944103a00 msgr2=0x7f0944103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:34.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.171+0000 7f0949000640 1 --2- 172.21.15.12:0/4206290989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0944103a00 0x7f0944103e20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0934009900 tx=0x7f093402bc40 comp rx=0 tx=0).stop 2024-03-17T23:54:34.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.171+0000 7f0949000640 1 -- 172.21.15.12:0/4206290989 shutdown_connections 2024-03-17T23:54:34.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.171+0000 7f0949000640 1 --2- 172.21.15.12:0/4206290989 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09441055b0 0x7f094410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:34.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.171+0000 7f0949000640 1 --2- 172.21.15.12:0/4206290989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0944104bf0 0x7f0944105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:34.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.171+0000 7f0949000640 1 --2- 172.21.15.12:0/4206290989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0944103a00 0x7f0944103e20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:34.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.172+0000 7f0949000640 1 -- 172.21.15.12:0/4206290989 >> 172.21.15.12:0/4206290989 conn(0x7f09440fef90 msgr2=0x7f09441013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:34.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.172+0000 7f0949000640 1 -- 172.21.15.12:0/4206290989 shutdown_connections 2024-03-17T23:54:34.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.172+0000 7f0949000640 1 -- 172.21.15.12:0/4206290989 wait complete. 2024-03-17T23:54:34.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.173+0000 7f0949000640 1 Processor -- start 2024-03-17T23:54:34.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.173+0000 7f0949000640 1 -- start start 2024-03-17T23:54:34.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.173+0000 7f0949000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0944103a00 0x7f0944199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:34.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.173+0000 7f0949000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0944104bf0 0x7f094419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:34.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.173+0000 7f0949000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09441055b0 0x7f09441a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:34.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.173+0000 7f0949000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f094410f6d0 con 0x7f0944104bf0 2024-03-17T23:54:34.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.173+0000 7f0949000640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f094410f550 con 0x7f09441055b0 2024-03-17T23:54:34.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.173+0000 7f0949000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f094410f850 con 0x7f0944103a00 2024-03-17T23:54:34.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.174+0000 7f0941e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0944103a00 0x7f0944199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:34.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.174+0000 7f0941e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0944103a00 0x7f0944199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:38732/0 (socket says 172.21.15.12:38732) 2024-03-17T23:54:34.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.174+0000 7f0941e00640 1 -- 172.21.15.12:0/4127777319 learned_addr learned my addr 172.21.15.12:0/4127777319 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:34.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.174+0000 7f0942800640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09441055b0 0x7f09441a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:34.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.174+0000 7f093be00640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0944104bf0 0x7f094419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:34.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.174+0000 7f0941e00640 1 -- 172.21.15.12:0/4127777319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09441055b0 msgr2=0x7f09441a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:34.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.174+0000 7f0941e00640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09441055b0 0x7f09441a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:34.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.174+0000 7f0941e00640 1 -- 172.21.15.12:0/4127777319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0944104bf0 msgr2=0x7f094419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:34.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.174+0000 7f0941e00640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0944104bf0 0x7f094419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:34.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.174+0000 7f0941e00640 1 -- 172.21.15.12:0/4127777319 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09441a1940 con 0x7f0944103a00 2024-03-17T23:54:34.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.174+0000 7f0942800640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09441055b0 0x7f09441a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:34.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.174+0000 7f093be00640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0944104bf0 0x7f094419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:34.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.174+0000 7f0941e00640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0944103a00 0x7f0944199d30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f093402efd0 tx=0x7f0934005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:34.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.175+0000 7f093aa00640 1 -- 172.21.15.12:0/4127777319 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0934041070 con 0x7f0944103a00 2024-03-17T23:54:34.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.175+0000 7f0949000640 1 -- 172.21.15.12:0/4127777319 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09441a1bd0 con 0x7f0944103a00 2024-03-17T23:54:34.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.175+0000 7f093aa00640 1 -- 172.21.15.12:0/4127777319 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09340044d0 con 0x7f0944103a00 2024-03-17T23:54:34.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.175+0000 7f093aa00640 1 -- 172.21.15.12:0/4127777319 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0934034440 con 0x7f0944103a00 2024-03-17T23:54:34.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.175+0000 7f0949000640 1 -- 172.21.15.12:0/4127777319 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09441a20e0 con 0x7f0944103a00 2024-03-17T23:54:34.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.177+0000 7f0949000640 1 -- 172.21.15.12:0/4127777319 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0944068a90 con 0x7f0944103a00 2024-03-17T23:54:34.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.178+0000 7f093aa00640 1 -- 172.21.15.12:0/4127777319 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0934044070 con 0x7f0944103a00 2024-03-17T23:54:34.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.179+0000 7f093aa00640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f09280a2800 0x7f09280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:34.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.180+0000 7f093aa00640 1 -- 172.21.15.12:0/4127777319 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f09340f0280 con 0x7f0944103a00 2024-03-17T23:54:34.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.180+0000 7f093be00640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f09280a2800 0x7f09280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:34.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.180+0000 7f093be00640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f09280a2800 0x7f09280a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f092c009fc0 tx=0x7f092c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.183+0000 7f093aa00640 1 -- 172.21.15.12:0/4127777319 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f09340bd0d0 con 0x7f0944103a00 2024-03-17T23:54:34.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.428+0000 7f0949000640 1 -- 172.21.15.12:0/4127777319 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09440623e0 con 0x7f0944103a00 2024-03-17T23:54:34.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.430+0000 7f093aa00640 1 -- 172.21.15.12:0/4127777319 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f09340c1170 con 0x7f0944103a00 2024-03-17T23:54:34.430 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:34.431 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-17T23:54:34.431 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 8 2024-03-17T23:54:34.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.432+0000 7f0949000640 1 -- 172.21.15.12:0/4127777319 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f09280a2800 msgr2=0x7f09280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:34.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.432+0000 7f0949000640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f09280a2800 0x7f09280a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f092c009fc0 tx=0x7f092c005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:34.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.432+0000 7f0949000640 1 -- 172.21.15.12:0/4127777319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0944103a00 msgr2=0x7f0944199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:34.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.432+0000 7f0949000640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0944103a00 0x7f0944199d30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f093402efd0 tx=0x7f0934005d50 comp rx=0 tx=0).stop 2024-03-17T23:54:34.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.432+0000 7f0949000640 1 -- 172.21.15.12:0/4127777319 shutdown_connections 2024-03-17T23:54:34.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.432+0000 7f0949000640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f09280a2800 0x7f09280a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:34.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.432+0000 7f0949000640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09441055b0 0x7f09441a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:34.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.432+0000 7f0949000640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0944104bf0 0x7f094419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.432+0000 7f0949000640 1 --2- 172.21.15.12:0/4127777319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0944103a00 0x7f0944199d30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.432+0000 7f0949000640 1 -- 172.21.15.12:0/4127777319 >> 172.21.15.12:0/4127777319 conn(0x7f09440fef90 msgr2=0x7f0944107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.433+0000 7f0949000640 1 -- 172.21.15.12:0/4127777319 shutdown_connections 2024-03-17T23:54:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:34.433+0000 7f0949000640 1 -- 172.21.15.12:0/4127777319 wait complete. 2024-03-17T23:54:34.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:34 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] up:boot 2024-03-17T23:54:34.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:34 smithi079 ceph-mon[36241]: fsmap 7 up:standby 2024-03-17T23:54:34.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:34 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-17T23:54:34.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:34 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:34.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:34 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:34.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:34 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:34.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:34 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:34.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:34 smithi079 ceph-mon[36241]: Deploying daemon mds.h on smithi079 2024-03-17T23:54:34.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:34 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/4127777319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:34.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:34 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] up:boot 2024-03-17T23:54:34.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:34 smithi191 ceph-mon[36708]: fsmap 7 up:standby 2024-03-17T23:54:34.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:34 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-17T23:54:34.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:34 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:34.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:34 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:34.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:34 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:34.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:34 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:34.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:34 smithi191 ceph-mon[36708]: Deploying daemon mds.h on smithi079 2024-03-17T23:54:34.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:34 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/4127777319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:34.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:34 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] up:boot 2024-03-17T23:54:34.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:34 smithi012 ceph-mon[31541]: fsmap 7 up:standby 2024-03-17T23:54:34.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:34 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-17T23:54:34.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:34 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:34.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:34 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:34.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:34 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:34.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:34 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:34.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:34 smithi012 ceph-mon[31541]: Deploying daemon mds.h on smithi079 2024-03-17T23:54:34.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:34 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4127777319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:35 smithi079 ceph-mon[36241]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:35.796 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 17 23:54:35 smithi079 systemd[1]: Starting Ceph mds.h for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-17T23:54:35.823 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:35 smithi012 ceph-mon[31541]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:35 smithi191 ceph-mon[36708]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:36.095 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 17 23:54:35 smithi079 podman[65945]: 2024-03-17 23:54:35.724049154 +0000 UTC m=+0.017757708 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-17T23:54:36.095 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 17 23:54:35 smithi079 podman[65945]: 2024-03-17 23:54:35.833753492 +0000 UTC m=+0.127462043 container create 7386f8afc90f03e7705193fd0d4da241f6427bf348b792ce8fecedd0107a92ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-h, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-17T23:54:36.095 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 17 23:54:36 smithi079 podman[65945]: 2024-03-17 23:54:36.095140499 +0000 UTC m=+0.388849046 container init 7386f8afc90f03e7705193fd0d4da241f6427bf348b792ce8fecedd0107a92ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-h, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-17T23:54:36.455 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:36.546 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 17 23:54:36 smithi079 podman[65945]: 2024-03-17 23:54:36.09781602 +0000 UTC m=+0.391524567 container start 7386f8afc90f03e7705193fd0d4da241f6427bf348b792ce8fecedd0107a92ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-h, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS) 2024-03-17T23:54:36.546 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 17 23:54:36 smithi079 ceph-mds[65964]: -- 172.21.15.79:0/4196712253 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563512b3fa40 con 0x563512b12800 2024-03-17T23:54:36.546 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 17 23:54:36 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-h[65960]: starting mds.h at 2024-03-17T23:54:36.546 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 17 23:54:36 smithi079 bash[65945]: 7386f8afc90f03e7705193fd0d4da241f6427bf348b792ce8fecedd0107a92ca 2024-03-17T23:54:36.546 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 17 23:54:36 smithi079 systemd[1]: Started Ceph mds.h for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:54:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.679+0000 7f0d8d400640 1 -- 172.21.15.12:0/2148264168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d881000c0 msgr2=0x7f0d881004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.679+0000 7f0d8d400640 1 --2- 172.21.15.12:0/2148264168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d881000c0 0x7f0d881004e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0d78009be0 tx=0x7f0d7802bf80 comp rx=0 tx=0).stop 2024-03-17T23:54:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.679+0000 7f0d8d400640 1 -- 172.21.15.12:0/2148264168 shutdown_connections 2024-03-17T23:54:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.679+0000 7f0d8d400640 1 --2- 172.21.15.12:0/2148264168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d881013c0 0x7f0d88101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.679+0000 7f0d8d400640 1 --2- 172.21.15.12:0/2148264168 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d88100a20 0x7f0d88100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.679+0000 7f0d8d400640 1 --2- 172.21.15.12:0/2148264168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d881000c0 0x7f0d881004e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:36.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.679+0000 7f0d8d400640 1 -- 172.21.15.12:0/2148264168 >> 172.21.15.12:0/2148264168 conn(0x7f0d880fbc40 msgr2=0x7f0d880fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:36.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.680+0000 7f0d8d400640 1 -- 172.21.15.12:0/2148264168 shutdown_connections 2024-03-17T23:54:36.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.680+0000 7f0d8d400640 1 -- 172.21.15.12:0/2148264168 wait complete. 2024-03-17T23:54:36.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.680+0000 7f0d8d400640 1 Processor -- start 2024-03-17T23:54:36.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.681+0000 7f0d8d400640 1 -- start start 2024-03-17T23:54:36.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.681+0000 7f0d8d400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d881000c0 0x7f0d88199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:36.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.681+0000 7f0d8d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d88100a20 0x7f0d8819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:36.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.681+0000 7f0d8d400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d881013c0 0x7f0d881a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:36.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.681+0000 7f0d8d400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8810f670 con 0x7f0d881000c0 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.681+0000 7f0d8d400640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8810f4f0 con 0x7f0d881013c0 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.681+0000 7f0d8d400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8810f7f0 con 0x7f0d88100a20 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.681+0000 7f0d86800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d881000c0 0x7f0d88199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.681+0000 7f0d7de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d88100a20 0x7f0d8819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.681+0000 7f0d86800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d881000c0 0x7f0d88199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47836/0 (socket says 172.21.15.12:47836) 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.681+0000 7f0d86800640 1 -- 172.21.15.12:0/2669774183 learned_addr learned my addr 172.21.15.12:0/2669774183 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.681+0000 7f0d7de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d88100a20 0x7f0d8819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:38742/0 (socket says 172.21.15.12:38742) 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.681+0000 7f0d87200640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d881013c0 0x7f0d881a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.682+0000 7f0d86800640 1 -- 172.21.15.12:0/2669774183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d88100a20 msgr2=0x7f0d8819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.682+0000 7f0d86800640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d88100a20 0x7f0d8819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.682+0000 7f0d86800640 1 -- 172.21.15.12:0/2669774183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d881013c0 msgr2=0x7f0d881a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.682+0000 7f0d86800640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d881013c0 0x7f0d881a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.682+0000 7f0d86800640 1 -- 172.21.15.12:0/2669774183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d881a18f0 con 0x7f0d881000c0 2024-03-17T23:54:36.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.682+0000 7f0d7de00640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d88100a20 0x7f0d8819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:36.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.682+0000 7f0d86800640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d881000c0 0x7f0d88199cb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0d7802c590 tx=0x7f0d78004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:36.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.682+0000 7f0d7fe00640 1 -- 172.21.15.12:0/2669774183 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d78043020 con 0x7f0d881000c0 2024-03-17T23:54:36.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.682+0000 7f0d7fe00640 1 -- 172.21.15.12:0/2669774183 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d78004670 con 0x7f0d881000c0 2024-03-17T23:54:36.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.682+0000 7f0d7fe00640 1 -- 172.21.15.12:0/2669774183 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7803e7d0 con 0x7f0d881000c0 2024-03-17T23:54:36.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.682+0000 7f0d8d400640 1 -- 172.21.15.12:0/2669774183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d881a1b80 con 0x7f0d881000c0 2024-03-17T23:54:36.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.683+0000 7f0d8d400640 1 -- 172.21.15.12:0/2669774183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d881a20d0 con 0x7f0d881000c0 2024-03-17T23:54:36.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.684+0000 7f0d8d400640 1 -- 172.21.15.12:0/2669774183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d8810afa0 con 0x7f0d881000c0 2024-03-17T23:54:36.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.686+0000 7f0d7fe00640 1 -- 172.21.15.12:0/2669774183 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0d78046020 con 0x7f0d881000c0 2024-03-17T23:54:36.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.688+0000 7f0d7fe00640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0d6c0a2800 0x7f0d6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:36.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.688+0000 7f0d7fe00640 1 -- 172.21.15.12:0/2669774183 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0d780f0db0 con 0x7f0d881000c0 2024-03-17T23:54:36.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.688+0000 7f0d7de00640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0d6c0a2800 0x7f0d6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:36.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.689+0000 7f0d7de00640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0d6c0a2800 0x7f0d6c0a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0d8819b1b0 tx=0x7f0d70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:36.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.693+0000 7f0d7fe00640 1 -- 172.21.15.12:0/2669774183 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0d7804d020 con 0x7f0d881000c0 2024-03-17T23:54:36.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.935+0000 7f0d8d400640 1 -- 172.21.15.12:0/2669774183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d880623e0 con 0x7f0d881000c0 2024-03-17T23:54:36.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.937+0000 7f0d7fe00640 1 -- 172.21.15.12:0/2669774183 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7f0d78004820 con 0x7f0d881000c0 2024-03-17T23:54:36.937 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:36.937 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-17T23:54:36.938 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 9 2024-03-17T23:54:36.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.939+0000 7f0d8d400640 1 -- 172.21.15.12:0/2669774183 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0d6c0a2800 msgr2=0x7f0d6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:36.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.939+0000 7f0d8d400640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0d6c0a2800 0x7f0d6c0a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0d8819b1b0 tx=0x7f0d70005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:36.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.939+0000 7f0d8d400640 1 -- 172.21.15.12:0/2669774183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d881000c0 msgr2=0x7f0d88199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:36.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.939+0000 7f0d8d400640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d881000c0 0x7f0d88199cb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0d7802c590 tx=0x7f0d78004270 comp rx=0 tx=0).stop 2024-03-17T23:54:36.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.939+0000 7f0d8d400640 1 -- 172.21.15.12:0/2669774183 shutdown_connections 2024-03-17T23:54:36.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.939+0000 7f0d8d400640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0d6c0a2800 0x7f0d6c0a4cc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:36.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.939+0000 7f0d8d400640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d881013c0 0x7f0d881a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:36.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.939+0000 7f0d8d400640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d88100a20 0x7f0d8819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:36.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.939+0000 7f0d8d400640 1 --2- 172.21.15.12:0/2669774183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d881000c0 0x7f0d88199cb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:36.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.940+0000 7f0d8d400640 1 -- 172.21.15.12:0/2669774183 >> 172.21.15.12:0/2669774183 conn(0x7f0d880fbc40 msgr2=0x7f0d880fd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:36.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.940+0000 7f0d8d400640 1 -- 172.21.15.12:0/2669774183 shutdown_connections 2024-03-17T23:54:36.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:36.940+0000 7f0d8d400640 1 -- 172.21.15.12:0/2669774183 wait complete. 2024-03-17T23:54:37.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:37 smithi012 ceph-mon[31541]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:37.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:37.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:37.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:37.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:37.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:37.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:37.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:37 smithi012 ceph-mon[31541]: Deploying daemon mds.i on smithi191 2024-03-17T23:54:37.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:37 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] up:boot 2024-03-17T23:54:37.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:37 smithi012 ceph-mon[31541]: fsmap 8 up:standby 2024-03-17T23:54:37.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:37 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-17T23:54:37.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:37 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2669774183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:37.485 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:37 smithi191 ceph-mon[36708]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:37.485 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:37 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:37.485 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:37 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:37.485 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:37 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:37.485 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:37 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:37.485 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:37 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:37.485 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:37 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:37.485 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:37 smithi191 ceph-mon[36708]: Deploying daemon mds.i on smithi191 2024-03-17T23:54:37.485 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:37 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] up:boot 2024-03-17T23:54:37.485 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:37 smithi191 ceph-mon[36708]: fsmap 8 up:standby 2024-03-17T23:54:37.486 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:37 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-17T23:54:37.486 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:37 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2669774183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:37.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:37 smithi079 ceph-mon[36241]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:37.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:37 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:37.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:37 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:37.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:37 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:37.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:37 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:37.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:37 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:37.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:37 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:37.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:37 smithi079 ceph-mon[36241]: Deploying daemon mds.i on smithi191 2024-03-17T23:54:37.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:37 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] up:boot 2024-03-17T23:54:37.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:37 smithi079 ceph-mon[36241]: fsmap 8 up:standby 2024-03-17T23:54:37.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:37 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-17T23:54:37.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:37 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2669774183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:38.753 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 17 23:54:38 smithi191 systemd[1]: Starting Ceph mds.i for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-17T23:54:38.959 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:39.041 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 17 23:54:38 smithi191 podman[67802]: 2024-03-17 23:54:38.655939433 +0000 UTC m=+0.018203281 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-17T23:54:39.042 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 17 23:54:38 smithi191 podman[67802]: 2024-03-17 23:54:38.789744047 +0000 UTC m=+0.152007909 container create 8439bf52950881deea7596b5c981f7eb814d2713d9c46d99e4b4f7c5903156ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-17T23:54:39.042 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 17 23:54:39 smithi191 podman[67802]: 2024-03-17 23:54:39.041576394 +0000 UTC m=+0.403840237 container init 8439bf52950881deea7596b5c981f7eb814d2713d9c46d99e4b4f7c5903156ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-i, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-17T23:54:39.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.186+0000 7f9063e00640 1 -- 172.21.15.12:0/823020837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9064103190 msgr2=0x7f9064109a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:39.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.186+0000 7f9063e00640 1 --2- 172.21.15.12:0/823020837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9064103190 0x7f9064109a20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f905c009900 tx=0x7f905c02bc30 comp rx=0 tx=0).stop 2024-03-17T23:54:39.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.186+0000 7f9063e00640 1 -- 172.21.15.12:0/823020837 shutdown_connections 2024-03-17T23:54:39.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.186+0000 7f9063e00640 1 --2- 172.21.15.12:0/823020837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9064103190 0x7f9064109a20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:39.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.186+0000 7f9063e00640 1 --2- 172.21.15.12:0/823020837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90641027d0 0x7f9064102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.186+0000 7f9063e00640 1 --2- 172.21.15.12:0/823020837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90641015e0 0x7f9064101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.186+0000 7f9063e00640 1 -- 172.21.15.12:0/823020837 >> 172.21.15.12:0/823020837 conn(0x7f90640fcb70 msgr2=0x7f90640fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.186+0000 7f9063e00640 1 -- 172.21.15.12:0/823020837 shutdown_connections 2024-03-17T23:54:39.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.186+0000 7f9063e00640 1 -- 172.21.15.12:0/823020837 wait complete. 2024-03-17T23:54:39.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.187+0000 7f9063e00640 1 Processor -- start 2024-03-17T23:54:39.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.187+0000 7f9063e00640 1 -- start start 2024-03-17T23:54:39.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9063e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90641015e0 0x7f9064199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:39.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9063e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90641027d0 0x7f9064199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:39.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9063e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064103190 0x7f90641a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:39.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9063e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f906410d290 con 0x7f9064103190 2024-03-17T23:54:39.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9063e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f906410d110 con 0x7f90641015e0 2024-03-17T23:54:39.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9063e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f906410d410 con 0x7f90641027d0 2024-03-17T23:54:39.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9063400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064103190 0x7f90641a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:39.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9063400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064103190 0x7f90641a1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47866/0 (socket says 172.21.15.12:47866) 2024-03-17T23:54:39.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9063400640 1 -- 172.21.15.12:0/4206932498 learned_addr learned my addr 172.21.15.12:0/4206932498 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:39.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9062a00640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90641015e0 0x7f9064199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:39.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9062000640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90641027d0 0x7f9064199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:39.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9062a00640 1 -- 172.21.15.12:0/4206932498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90641027d0 msgr2=0x7f9064199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:39.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9062a00640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90641027d0 0x7f9064199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:39.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9062a00640 1 -- 172.21.15.12:0/4206932498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064103190 msgr2=0x7f90641a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:39.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.188+0000 7f9062a00640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064103190 0x7f90641a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:39.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.189+0000 7f9062a00640 1 -- 172.21.15.12:0/4206932498 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90641a16b0 con 0x7f90641015e0 2024-03-17T23:54:39.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.189+0000 7f9063400640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064103190 0x7f90641a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:39.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.189+0000 7f9062a00640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90641015e0 0x7f9064199a90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f905400d6a0 tx=0x7f905400db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:39.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.189+0000 7f905b400640 1 -- 172.21.15.12:0/4206932498 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9054017070 con 0x7f90641015e0 2024-03-17T23:54:39.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.189+0000 7f9063e00640 1 -- 172.21.15.12:0/4206932498 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90641a19a0 con 0x7f90641015e0 2024-03-17T23:54:39.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.189+0000 7f905b400640 1 -- 172.21.15.12:0/4206932498 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9054009d00 con 0x7f90641015e0 2024-03-17T23:54:39.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.189+0000 7f905b400640 1 -- 172.21.15.12:0/4206932498 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9054003c60 con 0x7f90641015e0 2024-03-17T23:54:39.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.189+0000 7f9063e00640 1 -- 172.21.15.12:0/4206932498 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90641a1ef0 con 0x7f90641015e0 2024-03-17T23:54:39.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.190+0000 7f9063e00640 1 -- 172.21.15.12:0/4206932498 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9064068940 con 0x7f90641015e0 2024-03-17T23:54:39.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.192+0000 7f905b400640 1 -- 172.21.15.12:0/4206932498 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9054009820 con 0x7f90641015e0 2024-03-17T23:54:39.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.194+0000 7f905b400640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f90480a2800 0x7f90480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:39.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.194+0000 7f905b400640 1 -- 172.21.15.12:0/4206932498 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f90540cdf20 con 0x7f90641015e0 2024-03-17T23:54:39.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.194+0000 7f9062000640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f90480a2800 0x7f90480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:39.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.194+0000 7f9062000640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f90480a2800 0x7f90480a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f906419afa0 tx=0x7f904c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:39.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.198+0000 7f905b400640 1 -- 172.21.15.12:0/4206932498 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f905409ad70 con 0x7f90641015e0 2024-03-17T23:54:39.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:39 smithi191 ceph-mon[36708]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:39.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:39.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:39.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:39.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:39 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:39.342 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 17 23:54:39 smithi191 podman[67802]: 2024-03-17 23:54:39.044267813 +0000 UTC m=+0.406531651 container start 8439bf52950881deea7596b5c981f7eb814d2713d9c46d99e4b4f7c5903156ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-i, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-17T23:54:39.342 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 17 23:54:39 smithi191 ceph-mds[67821]: -- 172.21.15.191:0/2282017303 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555b58f17a40 con 0x555b58eea800 2024-03-17T23:54:39.342 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 17 23:54:39 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-i[67817]: starting mds.i at 2024-03-17T23:54:39.342 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 17 23:54:39 smithi191 bash[67802]: 8439bf52950881deea7596b5c981f7eb814d2713d9c46d99e4b4f7c5903156ca 2024-03-17T23:54:39.342 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 17 23:54:39 smithi191 systemd[1]: Started Ceph mds.i for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:54:39.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:39 smithi012 ceph-mon[31541]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:39.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:39.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:39.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:39.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:39 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.450+0000 7f9063e00640 1 -- 172.21.15.12:0/4206932498 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90641a28a0 con 0x7f90641015e0 2024-03-17T23:54:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.451+0000 7f905b400640 1 -- 172.21.15.12:0/4206932498 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7f905409ee10 con 0x7f90641015e0 2024-03-17T23:54:39.452 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:39.452 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-17T23:54:39.452 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 10 2024-03-17T23:54:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.453+0000 7f9063e00640 1 -- 172.21.15.12:0/4206932498 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f90480a2800 msgr2=0x7f90480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.453+0000 7f9063e00640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f90480a2800 0x7f90480a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f906419afa0 tx=0x7f904c009500 comp rx=0 tx=0).stop 2024-03-17T23:54:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.453+0000 7f9063e00640 1 -- 172.21.15.12:0/4206932498 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90641015e0 msgr2=0x7f9064199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.453+0000 7f9063e00640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90641015e0 0x7f9064199a90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f905400d6a0 tx=0x7f905400db60 comp rx=0 tx=0).stop 2024-03-17T23:54:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.453+0000 7f9063e00640 1 -- 172.21.15.12:0/4206932498 shutdown_connections 2024-03-17T23:54:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.454+0000 7f9063e00640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f90480a2800 0x7f90480a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.454+0000 7f9063e00640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064103190 0x7f90641a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.454+0000 7f9063e00640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90641027d0 0x7f9064199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.454+0000 7f9063e00640 1 --2- 172.21.15.12:0/4206932498 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90641015e0 0x7f9064199a90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.454+0000 7f9063e00640 1 -- 172.21.15.12:0/4206932498 >> 172.21.15.12:0/4206932498 conn(0x7f90640fcb70 msgr2=0x7f9064073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.454+0000 7f9063e00640 1 -- 172.21.15.12:0/4206932498 shutdown_connections 2024-03-17T23:54:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:39.454+0000 7f9063e00640 1 -- 172.21.15.12:0/4206932498 wait complete. 2024-03-17T23:54:39.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:39 smithi079 ceph-mon[36241]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:39.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:39.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:39.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:39.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:39 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:40.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:40 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:40.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:40 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:40.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:40 smithi079 ceph-mon[36241]: Deploying daemon mds.g on smithi012 2024-03-17T23:54:40.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:40 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] up:boot 2024-03-17T23:54:40.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:40 smithi079 ceph-mon[36241]: fsmap 9 up:standby 2024-03-17T23:54:40.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:40 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-17T23:54:40.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:40 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/4206932498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:40.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:40 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:40.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:40 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:40.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:40 smithi191 ceph-mon[36708]: Deploying daemon mds.g on smithi012 2024-03-17T23:54:40.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:40 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] up:boot 2024-03-17T23:54:40.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:40 smithi191 ceph-mon[36708]: fsmap 9 up:standby 2024-03-17T23:54:40.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:40 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-17T23:54:40.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:40 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/4206932498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:40.622 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:40 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:40.622 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:40 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:40.622 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:40 smithi012 ceph-mon[31541]: Deploying daemon mds.g on smithi012 2024-03-17T23:54:40.622 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:40 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] up:boot 2024-03-17T23:54:40.622 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:40 smithi012 ceph-mon[31541]: fsmap 9 up:standby 2024-03-17T23:54:40.622 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:40 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-17T23:54:40.622 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:40 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4206932498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:41.472 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:41.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:41 smithi079 ceph-mon[36241]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:41.566 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:41 smithi012 ceph-mon[31541]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:41.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:41 smithi191 ceph-mon[36708]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:41.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.708+0000 7f1194000640 1 -- 172.21.15.12:0/1085835065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f118c074240 msgr2=0x7f118c0746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:41.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.708+0000 7f1194000640 1 --2- 172.21.15.12:0/1085835065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f118c074240 0x7f118c0746c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1178009890 tx=0x7f117802bb70 comp rx=0 tx=0).stop 2024-03-17T23:54:41.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.708+0000 7f1194000640 1 -- 172.21.15.12:0/1085835065 shutdown_connections 2024-03-17T23:54:41.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.709+0000 7f1194000640 1 --2- 172.21.15.12:0/1085835065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f118c10ddc0 0x7f118c1101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:41.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.709+0000 7f1194000640 1 --2- 172.21.15.12:0/1085835065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f118c074240 0x7f118c0746c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:41.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.709+0000 7f1194000640 1 --2- 172.21.15.12:0/1085835065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f118c075820 0x7f118c073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:41.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.709+0000 7f1194000640 1 -- 172.21.15.12:0/1085835065 >> 172.21.15.12:0/1085835065 conn(0x7f118c0fcd80 msgr2=0x7f118c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:41.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.709+0000 7f1194000640 1 -- 172.21.15.12:0/1085835065 shutdown_connections 2024-03-17T23:54:41.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.709+0000 7f1194000640 1 -- 172.21.15.12:0/1085835065 wait complete. 2024-03-17T23:54:41.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.710+0000 7f1194000640 1 Processor -- start 2024-03-17T23:54:41.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.710+0000 7f1194000640 1 -- start start 2024-03-17T23:54:41.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.710+0000 7f1194000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f118c074240 0x7f118c19e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:41.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1194000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f118c075820 0x7f118c19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:41.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1194000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f118c10ddc0 0x7f118c1a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:41.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1194000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f118c113a60 con 0x7f118c074240 2024-03-17T23:54:41.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1194000640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f118c1138e0 con 0x7f118c10ddc0 2024-03-17T23:54:41.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1194000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f118c113be0 con 0x7f118c075820 2024-03-17T23:54:41.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1192000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f118c10ddc0 0x7f118c1a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:41.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1192000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f118c10ddc0 0x7f118c1a5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:44854/0 (socket says 172.21.15.12:44854) 2024-03-17T23:54:41.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1192000640 1 -- 172.21.15.12:0/729160789 learned_addr learned my addr 172.21.15.12:0/729160789 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:41.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1191600640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f118c074240 0x7f118c19e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:41.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1190c00640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f118c075820 0x7f118c19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:41.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1192000640 1 -- 172.21.15.12:0/729160789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f118c075820 msgr2=0x7f118c19e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:41.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1192000640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f118c075820 0x7f118c19e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:41.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1192000640 1 -- 172.21.15.12:0/729160789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f118c074240 msgr2=0x7f118c19e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:41.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1192000640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f118c074240 0x7f118c19e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:41.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1192000640 1 -- 172.21.15.12:0/729160789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f118c1a5d90 con 0x7f118c10ddc0 2024-03-17T23:54:41.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.711+0000 7f1190c00640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f118c075820 0x7f118c19e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:54:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.712+0000 7f1192000640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f118c10ddc0 0x7f118c1a5720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f118400e970 tx=0x7f118400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.712+0000 7f118a000640 1 -- 172.21.15.12:0/729160789 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118400ccc0 con 0x7f118c10ddc0 2024-03-17T23:54:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.712+0000 7f118a000640 1 -- 172.21.15.12:0/729160789 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1184005500 con 0x7f118c10ddc0 2024-03-17T23:54:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.712+0000 7f118a000640 1 -- 172.21.15.12:0/729160789 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1184010690 con 0x7f118c10ddc0 2024-03-17T23:54:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.712+0000 7f1194000640 1 -- 172.21.15.12:0/729160789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f118c1a6080 con 0x7f118c10ddc0 2024-03-17T23:54:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.712+0000 7f1194000640 1 -- 172.21.15.12:0/729160789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f118c1a6550 con 0x7f118c10ddc0 2024-03-17T23:54:41.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.713+0000 7f1194000640 1 -- 172.21.15.12:0/729160789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f118c103e80 con 0x7f118c10ddc0 2024-03-17T23:54:41.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.715+0000 7f118a000640 1 -- 172.21.15.12:0/729160789 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1184004e50 con 0x7f118c10ddc0 2024-03-17T23:54:41.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.717+0000 7f118a000640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f11700a2880 0x7f11700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:41.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.717+0000 7f118a000640 1 -- 172.21.15.12:0/729160789 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f11840ce920 con 0x7f118c10ddc0 2024-03-17T23:54:41.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.717+0000 7f1191600640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f11700a2880 0x7f11700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:41.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.718+0000 7f1191600640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f11700a2880 0x7f11700a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f117c005fd0 tx=0x7f117c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.719+0000 7f118a000640 1 -- 172.21.15.12:0/729160789 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f118409bdc0 con 0x7f118c10ddc0 2024-03-17T23:54:41.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.983+0000 7f1194000640 1 -- 172.21.15.12:0/729160789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f118c02d1b0 con 0x7f118c10ddc0 2024-03-17T23:54:41.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.984+0000 7f118a000640 1 -- 172.21.15.12:0/729160789 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7f118409b6c0 con 0x7f118c10ddc0 2024-03-17T23:54:41.985 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:41.985 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-17T23:54:41.985 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 10 2024-03-17T23:54:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.987+0000 7f1194000640 1 -- 172.21.15.12:0/729160789 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f11700a2880 msgr2=0x7f11700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.987+0000 7f1194000640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f11700a2880 0x7f11700a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f117c005fd0 tx=0x7f117c009500 comp rx=0 tx=0).stop 2024-03-17T23:54:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.987+0000 7f1194000640 1 -- 172.21.15.12:0/729160789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f118c10ddc0 msgr2=0x7f118c1a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.987+0000 7f1194000640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f118c10ddc0 0x7f118c1a5720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f118400e970 tx=0x7f118400ee30 comp rx=0 tx=0).stop 2024-03-17T23:54:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.987+0000 7f1194000640 1 -- 172.21.15.12:0/729160789 shutdown_connections 2024-03-17T23:54:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.987+0000 7f1194000640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f11700a2880 0x7f11700a4d40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.987+0000 7f1194000640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f118c10ddc0 0x7f118c1a5720 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.987+0000 7f1194000640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f118c075820 0x7f118c19e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.987+0000 7f1194000640 1 --2- 172.21.15.12:0/729160789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f118c074240 0x7f118c19e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.987+0000 7f1194000640 1 -- 172.21.15.12:0/729160789 >> 172.21.15.12:0/729160789 conn(0x7f118c0fcd80 msgr2=0x7f118c101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.987+0000 7f1194000640 1 -- 172.21.15.12:0/729160789 shutdown_connections 2024-03-17T23:54:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:41.987+0000 7f1194000640 1 -- 172.21.15.12:0/729160789 wait complete. 2024-03-17T23:54:42.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:42 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/729160789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:42.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:42 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/729160789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:42.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:42 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/729160789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:42.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 17 23:54:42 smithi012 systemd[1]: Starting Ceph mds.g for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-17T23:54:43.141 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 17 23:54:42 smithi012 podman[70074]: 2024-03-17 23:54:42.738202203 +0000 UTC m=+0.034952225 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-17T23:54:43.142 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 17 23:54:42 smithi012 podman[70074]: 2024-03-17 23:54:42.843006055 +0000 UTC m=+0.139756115 container create 2c8d7200ec4c560929c932af8a4eb029e7324180ad1083b1b79c90926e70aa87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-g, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-17T23:54:43.142 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 17 23:54:43 smithi012 podman[70074]: 2024-03-17 23:54:43.139277324 +0000 UTC m=+0.436027387 container init 2c8d7200ec4c560929c932af8a4eb029e7324180ad1083b1b79c90926e70aa87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-g, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-17T23:54:43.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:43 smithi012 ceph-mon[31541]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:43.419 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 17 23:54:43 smithi012 podman[70074]: 2024-03-17 23:54:43.151042422 +0000 UTC m=+0.447792461 container start 2c8d7200ec4c560929c932af8a4eb029e7324180ad1083b1b79c90926e70aa87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-17T23:54:43.419 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 17 23:54:43 smithi012 ceph-mds[70093]: -- 172.21.15.12:0/1008366618 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fdb9133a40 con 0x55fdb9106c00 2024-03-17T23:54:43.419 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 17 23:54:43 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-g[70089]: starting mds.g at 2024-03-17T23:54:43.420 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 17 23:54:43 smithi012 bash[70074]: 2c8d7200ec4c560929c932af8a4eb029e7324180ad1083b1b79c90926e70aa87 2024-03-17T23:54:43.420 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 17 23:54:43 smithi012 systemd[1]: Started Ceph mds.g for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:54:43.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:43 smithi079 ceph-mon[36241]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:43.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:43 smithi191 ceph-mon[36708]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:44.007 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:44.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.222+0000 7fc52ec00640 1 -- 172.21.15.12:0/4289629072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5281049a0 msgr2=0x7fc528106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:44.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.222+0000 7fc52ec00640 1 --2- 172.21.15.12:0/4289629072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5281049a0 0x7fc528106d90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc514009900 tx=0x7fc51402bc40 comp rx=0 tx=0).stop 2024-03-17T23:54:44.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.222+0000 7fc52ec00640 1 -- 172.21.15.12:0/4289629072 shutdown_connections 2024-03-17T23:54:44.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.222+0000 7fc52ec00640 1 --2- 172.21.15.12:0/4289629072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5281072d0 0x7fc528109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:44.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.222+0000 7fc52ec00640 1 --2- 172.21.15.12:0/4289629072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5281049a0 0x7fc528106d90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:44.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.222+0000 7fc52ec00640 1 --2- 172.21.15.12:0/4289629072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc528102070 0x7fc528104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:44.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.222+0000 7fc52ec00640 1 -- 172.21.15.12:0/4289629072 >> 172.21.15.12:0/4289629072 conn(0x7fc5280fba60 msgr2=0x7fc5280fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:44.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.223+0000 7fc52ec00640 1 -- 172.21.15.12:0/4289629072 shutdown_connections 2024-03-17T23:54:44.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.223+0000 7fc52ec00640 1 -- 172.21.15.12:0/4289629072 wait complete. 2024-03-17T23:54:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.223+0000 7fc52ec00640 1 Processor -- start 2024-03-17T23:54:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.224+0000 7fc52ec00640 1 -- start start 2024-03-17T23:54:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.224+0000 7fc52ec00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc528102070 0x7fc528199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.224+0000 7fc52ec00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5281049a0 0x7fc52819a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.224+0000 7fc52ec00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5281072d0 0x7fc5281a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.224+0000 7fc52ec00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc52810d2c0 con 0x7fc5281049a0 2024-03-17T23:54:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.224+0000 7fc52ec00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc52810d140 con 0x7fc528102070 2024-03-17T23:54:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.224+0000 7fc52ec00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc52810d440 con 0x7fc5281072d0 2024-03-17T23:54:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.224+0000 7fc52ce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5281049a0 0x7fc52819a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.224+0000 7fc52d800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc528102070 0x7fc528199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.224+0000 7fc52e200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5281072d0 0x7fc5281a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.224+0000 7fc52d800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc528102070 0x7fc528199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:51008/0 (socket says 172.21.15.12:51008) 2024-03-17T23:54:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.224+0000 7fc52d800640 1 -- 172.21.15.12:0/1759133324 learned_addr learned my addr 172.21.15.12:0/1759133324 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.225+0000 7fc52e200640 1 -- 172.21.15.12:0/1759133324 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc528102070 msgr2=0x7fc528199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.225+0000 7fc52e200640 1 --2- 172.21.15.12:0/1759133324 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc528102070 0x7fc528199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.225+0000 7fc52e200640 1 -- 172.21.15.12:0/1759133324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5281049a0 msgr2=0x7fc52819a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.225+0000 7fc52e200640 1 --2- 172.21.15.12:0/1759133324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5281049a0 0x7fc52819a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.225+0000 7fc52e200640 1 -- 172.21.15.12:0/1759133324 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5281a1750 con 0x7fc5281072d0 2024-03-17T23:54:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.225+0000 7fc52e200640 1 --2- 172.21.15.12:0/1759133324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5281072d0 0x7fc5281a10e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc52000ec50 tx=0x7fc52000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.225+0000 7fc526000640 1 -- 172.21.15.12:0/1759133324 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc520003b70 con 0x7fc5281072d0 2024-03-17T23:54:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.225+0000 7fc526000640 1 -- 172.21.15.12:0/1759133324 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc520003d10 con 0x7fc5281072d0 2024-03-17T23:54:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.225+0000 7fc526000640 1 -- 172.21.15.12:0/1759133324 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc520010890 con 0x7fc5281072d0 2024-03-17T23:54:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.225+0000 7fc52ec00640 1 -- 172.21.15.12:0/1759133324 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5281a1a40 con 0x7fc5281072d0 2024-03-17T23:54:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.226+0000 7fc52ec00640 1 -- 172.21.15.12:0/1759133324 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5281a1f10 con 0x7fc5281072d0 2024-03-17T23:54:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.227+0000 7fc52ec00640 1 -- 172.21.15.12:0/1759133324 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc528108e20 con 0x7fc5281072d0 2024-03-17T23:54:44.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.229+0000 7fc526000640 1 -- 172.21.15.12:0/1759133324 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc520003fc0 con 0x7fc5281072d0 2024-03-17T23:54:44.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.230+0000 7fc526000640 1 --2- 172.21.15.12:0/1759133324 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc5100a2800 0x7fc5100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:44.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.230+0000 7fc526000640 1 -- 172.21.15.12:0/1759133324 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc5200ceb40 con 0x7fc5281072d0 2024-03-17T23:54:44.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.230+0000 7fc52d800640 1 --2- 172.21.15.12:0/1759133324 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc5100a2800 0x7fc5100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:44.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.231+0000 7fc52d800640 1 --2- 172.21.15.12:0/1759133324 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc5100a2800 0x7fc5100a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009cd0 tx=0x7fc51c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.234+0000 7fc526000640 1 -- 172.21.15.12:0/1759133324 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc52009c080 con 0x7fc5281072d0 2024-03-17T23:54:44.417 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:44 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.12:6838/1516405819,v1:172.21.15.12:6839/1516405819] up:boot 2024-03-17T23:54:44.417 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:44 smithi012 ceph-mon[31541]: fsmap 10 up:standby 2024-03-17T23:54:44.417 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-17T23:54:44.417 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:44.417 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:44.417 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:44.417 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:44.417 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:44.417 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:44 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:44.417 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:44 smithi012 ceph-mon[31541]: Deploying daemon mds.a on smithi012 2024-03-17T23:54:44.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.495+0000 7fc52ec00640 1 -- 172.21.15.12:0/1759133324 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc52802d380 con 0x7fc5281072d0 2024-03-17T23:54:44.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.496+0000 7fc526000640 1 -- 172.21.15.12:0/1759133324 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7fc52009b980 con 0x7fc5281072d0 2024-03-17T23:54:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:44.497 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-17T23:54:44.497 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 11 2024-03-17T23:54:44.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.499+0000 7fc52ec00640 1 -- 172.21.15.12:0/1759133324 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc5100a2800 msgr2=0x7fc5100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:44.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.499+0000 7fc52ec00640 1 --2- 172.21.15.12:0/1759133324 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc5100a2800 0x7fc5100a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009cd0 tx=0x7fc51c005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:44.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.499+0000 7fc52ec00640 1 -- 172.21.15.12:0/1759133324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5281072d0 msgr2=0x7fc5281a10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:44.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.499+0000 7fc52ec00640 1 --2- 172.21.15.12:0/1759133324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5281072d0 0x7fc5281a10e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc52000ec50 tx=0x7fc52000c6a0 comp rx=0 tx=0).stop 2024-03-17T23:54:44.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.499+0000 7fc52ec00640 1 -- 172.21.15.12:0/1759133324 shutdown_connections 2024-03-17T23:54:44.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.499+0000 7fc52ec00640 1 --2- 172.21.15.12:0/1759133324 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc5100a2800 0x7fc5100a4cc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:44.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.499+0000 7fc52ec00640 1 --2- 172.21.15.12:0/1759133324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5281072d0 0x7fc5281a10e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:44.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.499+0000 7fc52ec00640 1 --2- 172.21.15.12:0/1759133324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5281049a0 0x7fc52819a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:44.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.499+0000 7fc52ec00640 1 --2- 172.21.15.12:0/1759133324 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc528102070 0x7fc528199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:44.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.499+0000 7fc52ec00640 1 -- 172.21.15.12:0/1759133324 >> 172.21.15.12:0/1759133324 conn(0x7fc5280fba60 msgr2=0x7fc5281016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:44.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.499+0000 7fc52ec00640 1 -- 172.21.15.12:0/1759133324 shutdown_connections 2024-03-17T23:54:44.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:44.499+0000 7fc52ec00640 1 -- 172.21.15.12:0/1759133324 wait complete. 2024-03-17T23:54:44.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:44 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.12:6838/1516405819,v1:172.21.15.12:6839/1516405819] up:boot 2024-03-17T23:54:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:44 smithi079 ceph-mon[36241]: fsmap 10 up:standby 2024-03-17T23:54:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-17T23:54:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:44 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:44.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:44 smithi079 ceph-mon[36241]: Deploying daemon mds.a on smithi012 2024-03-17T23:54:44.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:44 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.12:6838/1516405819,v1:172.21.15.12:6839/1516405819] up:boot 2024-03-17T23:54:44.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:44 smithi191 ceph-mon[36708]: fsmap 10 up:standby 2024-03-17T23:54:44.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-17T23:54:44.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:44.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:44.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:44.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:44.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:44.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:44 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:44.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:44 smithi191 ceph-mon[36708]: Deploying daemon mds.a on smithi012 2024-03-17T23:54:45.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:45 smithi012 ceph-mon[31541]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:45.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:45 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1759133324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:45.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:45 smithi079 ceph-mon[36241]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:45.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:45 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1759133324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:45.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:45 smithi191 ceph-mon[36708]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:45.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:45 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1759133324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:46.519 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:46.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.740+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2403444874 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7a4105180 msgr2=0x7fd7a4105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:46.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.740+0000 7fd7a3e00640 1 --2- 172.21.15.12:0/2403444874 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7a4105180 0x7fd7a4105600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd79c009ca0 tx=0x7fd79c02c0f0 comp rx=0 tx=0).stop 2024-03-17T23:54:46.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.740+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2403444874 shutdown_connections 2024-03-17T23:54:46.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.740+0000 7fd7a3e00640 1 --2- 172.21.15.12:0/2403444874 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7a4105180 0x7fd7a4105600 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:46.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.740+0000 7fd7a3e00640 1 --2- 172.21.15.12:0/2403444874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7a41027d0 0x7fd7a4104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:46.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.740+0000 7fd7a3e00640 1 --2- 172.21.15.12:0/2403444874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7a40ffe60 0x7fd7a4102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:46.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.740+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2403444874 >> 172.21.15.12:0/2403444874 conn(0x7fd7a40fba60 msgr2=0x7fd7a40fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:46.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.741+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2403444874 shutdown_connections 2024-03-17T23:54:46.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.741+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2403444874 wait complete. 2024-03-17T23:54:46.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.742+0000 7fd7a3e00640 1 Processor -- start 2024-03-17T23:54:46.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.742+0000 7fd7a3e00640 1 -- start start 2024-03-17T23:54:46.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.742+0000 7fd7a3e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7a40ffe60 0x7fd7a419de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:46.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.742+0000 7fd7a3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7a41027d0 0x7fd7a419e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:46.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.742+0000 7fd7a3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7a4105180 0x7fd7a41a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:46.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.742+0000 7fd7a3e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a4113860 con 0x7fd7a41027d0 2024-03-17T23:54:46.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.742+0000 7fd7a3e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a41136e0 con 0x7fd7a40ffe60 2024-03-17T23:54:46.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.742+0000 7fd7a3e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a41139e0 con 0x7fd7a4105180 2024-03-17T23:54:46.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.742+0000 7fd7a2a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7a40ffe60 0x7fd7a419de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:46.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.742+0000 7fd7a2a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7a40ffe60 0x7fd7a419de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:51016/0 (socket says 172.21.15.12:51016) 2024-03-17T23:54:46.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.742+0000 7fd7a2a00640 1 -- 172.21.15.12:0/2378410698 learned_addr learned my addr 172.21.15.12:0/2378410698 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:46.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.742+0000 7fd79be00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7a41027d0 0x7fd7a419e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:46.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.743+0000 7fd7a3400640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7a4105180 0x7fd7a41a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:46.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.743+0000 7fd7a2a00640 1 -- 172.21.15.12:0/2378410698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7a4105180 msgr2=0x7fd7a41a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:46.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.743+0000 7fd7a2a00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7a4105180 0x7fd7a41a53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:46.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.743+0000 7fd7a2a00640 1 -- 172.21.15.12:0/2378410698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7a41027d0 msgr2=0x7fd7a419e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:46.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.743+0000 7fd7a2a00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7a41027d0 0x7fd7a419e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:46.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.743+0000 7fd7a2a00640 1 -- 172.21.15.12:0/2378410698 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a41a5a50 con 0x7fd7a40ffe60 2024-03-17T23:54:46.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.743+0000 7fd7a3400640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7a4105180 0x7fd7a41a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:54:46.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.743+0000 7fd79be00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7a41027d0 0x7fd7a419e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:54:46.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.743+0000 7fd7a2a00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7a40ffe60 0x7fd7a419de30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd79400d950 tx=0x7fd79400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:46.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.743+0000 7fd7a8a00640 1 -- 172.21.15.12:0/2378410698 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794017030 con 0x7fd7a40ffe60 2024-03-17T23:54:46.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.743+0000 7fd7a8a00640 1 -- 172.21.15.12:0/2378410698 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd794004020 con 0x7fd7a40ffe60 2024-03-17T23:54:46.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.743+0000 7fd7a8a00640 1 -- 172.21.15.12:0/2378410698 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7940047a0 con 0x7fd7a40ffe60 2024-03-17T23:54:46.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.743+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2378410698 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a41a5d40 con 0x7fd7a40ffe60 2024-03-17T23:54:46.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.744+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2378410698 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a41a6220 con 0x7fd7a40ffe60 2024-03-17T23:54:46.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.745+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2378410698 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7a40fd550 con 0x7fd7a40ffe60 2024-03-17T23:54:46.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.748+0000 7fd7a8a00640 1 -- 172.21.15.12:0/2378410698 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd7940041d0 con 0x7fd7a40ffe60 2024-03-17T23:54:46.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.749+0000 7fd7a8a00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd7880a2880 0x7fd7880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:46.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.749+0000 7fd7a8a00640 1 -- 172.21.15.12:0/2378410698 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd7940ce0f0 con 0x7fd7a40ffe60 2024-03-17T23:54:46.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.750+0000 7fd79be00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd7880a2880 0x7fd7880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:46.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.750+0000 7fd79be00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd7880a2880 0x7fd7880a4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd7a419f340 tx=0x7fd78c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:46.754+0000 7fd7a8a00640 1 -- 172.21.15.12:0/2378410698 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd79409b590 con 0x7fd7a40ffe60 2024-03-17T23:54:46.919 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 17 23:54:46 smithi012 systemd[1]: Starting Ceph mds.a for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-17T23:54:47.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.032+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2378410698 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7a402d1b0 con 0x7fd7a40ffe60 2024-03-17T23:54:47.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.033+0000 7fd7a8a00640 1 -- 172.21.15.12:0/2378410698 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7fd79409ae90 con 0x7fd7a40ffe60 2024-03-17T23:54:47.034 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:47.034 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-17T23:54:47.035 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 11 2024-03-17T23:54:47.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.036+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2378410698 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd7880a2880 msgr2=0x7fd7880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:47.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.036+0000 7fd7a3e00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd7880a2880 0x7fd7880a4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd7a419f340 tx=0x7fd78c005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:47.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.036+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2378410698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7a40ffe60 msgr2=0x7fd7a419de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:47.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.036+0000 7fd7a3e00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7a40ffe60 0x7fd7a419de30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd79400d950 tx=0x7fd79400de10 comp rx=0 tx=0).stop 2024-03-17T23:54:47.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.036+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2378410698 shutdown_connections 2024-03-17T23:54:47.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.036+0000 7fd7a3e00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd7880a2880 0x7fd7880a4d40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:47.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.036+0000 7fd7a3e00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7a4105180 0x7fd7a41a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:47.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.036+0000 7fd7a3e00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7a41027d0 0x7fd7a419e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.036+0000 7fd7a3e00640 1 --2- 172.21.15.12:0/2378410698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7a40ffe60 0x7fd7a419de30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.036+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2378410698 >> 172.21.15.12:0/2378410698 conn(0x7fd7a40fba60 msgr2=0x7fd7a41007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.036+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2378410698 shutdown_connections 2024-03-17T23:54:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:47.036+0000 7fd7a3e00640 1 -- 172.21.15.12:0/2378410698 wait complete. 2024-03-17T23:54:47.339 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 17 23:54:47 smithi012 podman[70452]: 2024-03-17 23:54:46.927386186 +0000 UTC m=+0.040769191 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-17T23:54:47.339 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 17 23:54:47 smithi012 podman[70452]: 2024-03-17 23:54:47.058531183 +0000 UTC m=+0.171914225 container create 1d70a05843c8fc5aa848cdbb247928db76b1b92c4d6a4630a98a725e7648e472 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-a, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-17T23:54:47.339 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 17 23:54:47 smithi012 podman[70452]: 2024-03-17 23:54:47.337004869 +0000 UTC m=+0.450387918 container init 1d70a05843c8fc5aa848cdbb247928db76b1b92c4d6a4630a98a725e7648e472 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-17T23:54:47.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:47 smithi012 ceph-mon[31541]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:47.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:47 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2378410698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:47.669 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 17 23:54:47 smithi012 podman[70452]: 2024-03-17 23:54:47.348152972 +0000 UTC m=+0.461536001 container start 1d70a05843c8fc5aa848cdbb247928db76b1b92c4d6a4630a98a725e7648e472 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-a, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-17T23:54:47.670 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 17 23:54:47 smithi012 ceph-mds[70473]: -- 172.21.15.12:0/3871437615 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5654f5bb5a40 con 0x5654f5b88c00 2024-03-17T23:54:47.670 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 17 23:54:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-a[70469]: starting mds.a at 2024-03-17T23:54:47.670 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 17 23:54:47 smithi012 bash[70452]: 1d70a05843c8fc5aa848cdbb247928db76b1b92c4d6a4630a98a725e7648e472 2024-03-17T23:54:47.670 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 17 23:54:47 smithi012 systemd[1]: Started Ceph mds.a for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:54:47.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:47 smithi079 ceph-mon[36241]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:47.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:47 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2378410698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:47.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:47 smithi191 ceph-mon[36708]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:47.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:47 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2378410698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:48.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:48 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] up:boot 2024-03-17T23:54:48.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:48 smithi079 ceph-mon[36241]: fsmap 11 up:standby 2024-03-17T23:54:48.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-17T23:54:48.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:48.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:48.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:48.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:48.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:48.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:48.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:48 smithi079 ceph-mon[36241]: Deploying daemon mds.b on smithi079 2024-03-17T23:54:48.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:48 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] up:boot 2024-03-17T23:54:48.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:48 smithi012 ceph-mon[31541]: fsmap 11 up:standby 2024-03-17T23:54:48.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-17T23:54:48.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:48.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:48.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:48.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:48.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:48.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:48.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:48 smithi012 ceph-mon[31541]: Deploying daemon mds.b on smithi079 2024-03-17T23:54:48.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:48 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] up:boot 2024-03-17T23:54:48.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:48 smithi191 ceph-mon[36708]: fsmap 11 up:standby 2024-03-17T23:54:48.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-17T23:54:48.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:48.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:48.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:48.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-17T23:54:48.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-17T23:54:48.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:48.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:48 smithi191 ceph-mon[36708]: Deploying daemon mds.b on smithi079 2024-03-17T23:54:49.058 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:49.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.294+0000 7f0385a00640 1 -- 172.21.15.12:0/1410551379 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03800773e0 msgr2=0x7f0380077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:49.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.294+0000 7f0385a00640 1 --2- 172.21.15.12:0/1410551379 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03800773e0 0x7f0380077840 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f036c009860 tx=0x7f036c02bb70 comp rx=0 tx=0).stop 2024-03-17T23:54:49.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.294+0000 7f0385a00640 1 -- 172.21.15.12:0/1410551379 shutdown_connections 2024-03-17T23:54:49.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.294+0000 7f0385a00640 1 --2- 172.21.15.12:0/1410551379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0380077d80 0x7f0380078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:49.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.294+0000 7f0385a00640 1 --2- 172.21.15.12:0/1410551379 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03800773e0 0x7f0380077840 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:49.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.294+0000 7f0385a00640 1 --2- 172.21.15.12:0/1410551379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03800761a0 0x7f03800765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:49.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.294+0000 7f0385a00640 1 -- 172.21.15.12:0/1410551379 >> 172.21.15.12:0/1410551379 conn(0x7f03800fedc0 msgr2=0x7f03801011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:49.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.294+0000 7f0385a00640 1 -- 172.21.15.12:0/1410551379 shutdown_connections 2024-03-17T23:54:49.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.294+0000 7f0385a00640 1 -- 172.21.15.12:0/1410551379 wait complete. 2024-03-17T23:54:49.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.295+0000 7f0385a00640 1 Processor -- start 2024-03-17T23:54:49.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.295+0000 7f0385a00640 1 -- start start 2024-03-17T23:54:49.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.295+0000 7f0385a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03800761a0 0x7f038019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:49.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.295+0000 7f0385a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03800773e0 0x7f038019e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:49.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.295+0000 7f0385a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0380077d80 0x7f03801a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:49.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.295+0000 7f0385a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f03801138c0 con 0x7f03800773e0 2024-03-17T23:54:49.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.295+0000 7f0385a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0380113740 con 0x7f03800761a0 2024-03-17T23:54:49.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.295+0000 7f0385a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0380113a40 con 0x7f0380077d80 2024-03-17T23:54:49.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.296+0000 7f037f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03800773e0 0x7f038019e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:49.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.296+0000 7f037f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03800773e0 0x7f038019e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52636/0 (socket says 172.21.15.12:52636) 2024-03-17T23:54:49.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.296+0000 7f037f400640 1 -- 172.21.15.12:0/1338889312 learned_addr learned my addr 172.21.15.12:0/1338889312 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:49.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.296+0000 7f0385000640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0380077d80 0x7f03801a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:49.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.296+0000 7f037fe00640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03800761a0 0x7f038019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:49.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.296+0000 7f037f400640 1 -- 172.21.15.12:0/1338889312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0380077d80 msgr2=0x7f03801a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:49.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.296+0000 7f037f400640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0380077d80 0x7f03801a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:49.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.296+0000 7f037f400640 1 -- 172.21.15.12:0/1338889312 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03800761a0 msgr2=0x7f038019de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:49.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.296+0000 7f037f400640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03800761a0 0x7f038019de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:49.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.296+0000 7f037f400640 1 -- 172.21.15.12:0/1338889312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03801a5ab0 con 0x7f03800773e0 2024-03-17T23:54:49.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.296+0000 7f037f400640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03800773e0 0x7f038019e3d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f036c0349e0 tx=0x7f036c032450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:49.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.297+0000 7f037cc00640 1 -- 172.21.15.12:0/1338889312 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c041030 con 0x7f03800773e0 2024-03-17T23:54:49.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.297+0000 7f0385a00640 1 -- 172.21.15.12:0/1338889312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03801a5d40 con 0x7f03800773e0 2024-03-17T23:54:49.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.297+0000 7f037cc00640 1 -- 172.21.15.12:0/1338889312 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f036c02c510 con 0x7f03800773e0 2024-03-17T23:54:49.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.297+0000 7f037cc00640 1 -- 172.21.15.12:0/1338889312 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c039040 con 0x7f03800773e0 2024-03-17T23:54:49.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.297+0000 7f0385a00640 1 -- 172.21.15.12:0/1338889312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03801a61d0 con 0x7f03800773e0 2024-03-17T23:54:49.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.298+0000 7f0385a00640 1 -- 172.21.15.12:0/1338889312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03801101a0 con 0x7f03800773e0 2024-03-17T23:54:49.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.300+0000 7f037cc00640 1 -- 172.21.15.12:0/1338889312 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f036c044070 con 0x7f03800773e0 2024-03-17T23:54:49.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.301+0000 7f037cc00640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f03640a2800 0x7f03640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:49.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.302+0000 7f037cc00640 1 -- 172.21.15.12:0/1338889312 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f036c0efb20 con 0x7f03800773e0 2024-03-17T23:54:49.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.302+0000 7f037fe00640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f03640a2800 0x7f03640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:49.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.302+0000 7f037fe00640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f03640a2800 0x7f03640a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0370009bd0 tx=0x7f0370005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:49.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.306+0000 7f037cc00640 1 -- 172.21.15.12:0/1338889312 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f036c0bd060 con 0x7f03800773e0 2024-03-17T23:54:49.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.551+0000 7f0385a00640 1 -- 172.21.15.12:0/1338889312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f038002d3d0 con 0x7f03800773e0 2024-03-17T23:54:49.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.552+0000 7f037cc00640 1 -- 172.21.15.12:0/1338889312 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7f036c0bc960 con 0x7f03800773e0 2024-03-17T23:54:49.553 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:49.553 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/2073607248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":2073607248},{"type":"v1","addr":"172.21.15.12:6841","nonce":2073607248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-03-17T23:54:49.553 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 12 2024-03-17T23:54:49.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.555+0000 7f0385a00640 1 -- 172.21.15.12:0/1338889312 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f03640a2800 msgr2=0x7f03640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:49.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.555+0000 7f0385a00640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f03640a2800 0x7f03640a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0370009bd0 tx=0x7f0370005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:49.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.555+0000 7f0385a00640 1 -- 172.21.15.12:0/1338889312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03800773e0 msgr2=0x7f038019e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:49.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.555+0000 7f0385a00640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03800773e0 0x7f038019e3d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f036c0349e0 tx=0x7f036c032450 comp rx=0 tx=0).stop 2024-03-17T23:54:49.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.555+0000 7f0385a00640 1 -- 172.21.15.12:0/1338889312 shutdown_connections 2024-03-17T23:54:49.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.555+0000 7f0385a00640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f03640a2800 0x7f03640a4cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:49.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.555+0000 7f0385a00640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0380077d80 0x7f03801a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:49.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.555+0000 7f0385a00640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03800773e0 0x7f038019e3d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:49.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.555+0000 7f0385a00640 1 --2- 172.21.15.12:0/1338889312 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03800761a0 0x7f038019de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:49.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.555+0000 7f0385a00640 1 -- 172.21.15.12:0/1338889312 >> 172.21.15.12:0/1338889312 conn(0x7f03800fedc0 msgr2=0x7f03801008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:49.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.555+0000 7f0385a00640 1 -- 172.21.15.12:0/1338889312 shutdown_connections 2024-03-17T23:54:49.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:49.555+0000 7f0385a00640 1 -- 172.21.15.12:0/1338889312 wait complete. 2024-03-17T23:54:49.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:49 smithi012 ceph-mon[31541]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:49.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:49 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1338889312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:50.024 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:49 smithi079 ceph-mon[36241]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:50.024 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:49 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1338889312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:50.025 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 17 23:54:49 smithi079 systemd[1]: Starting Ceph mds.b for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-17T23:54:50.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:49 smithi191 ceph-mon[36708]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:50.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:49 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1338889312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:50.291 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 17 23:54:50 smithi079 podman[66269]: 2024-03-17 23:54:49.927048333 +0000 UTC m=+0.017698145 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-17T23:54:50.291 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 17 23:54:50 smithi079 podman[66269]: 2024-03-17 23:54:50.054723791 +0000 UTC m=+0.145373585 container create 90b5110dc80e26e26c77d1494b5f0767e7d7ca78eb78fecbaf2d033a257dfc85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-17T23:54:50.291 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 17 23:54:50 smithi079 podman[66269]: 2024-03-17 23:54:50.265555156 +0000 UTC m=+0.356204951 container init 90b5110dc80e26e26c77d1494b5f0767e7d7ca78eb78fecbaf2d033a257dfc85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-17T23:54:50.291 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 17 23:54:50 smithi079 podman[66269]: 2024-03-17 23:54:50.268163287 +0000 UTC m=+0.358813084 container start 90b5110dc80e26e26c77d1494b5f0767e7d7ca78eb78fecbaf2d033a257dfc85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-17T23:54:50.291 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 17 23:54:50 smithi079 ceph-mds[66289]: -- 172.21.15.79:0/2184219968 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55de07b03a40 con 0x55de07ad6c00 2024-03-17T23:54:50.545 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 17 23:54:50 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b[66285]: starting mds.b at 2024-03-17T23:54:50.546 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 17 23:54:50 smithi079 bash[66269]: 90b5110dc80e26e26c77d1494b5f0767e7d7ca78eb78fecbaf2d033a257dfc85 2024-03-17T23:54:50.546 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 17 23:54:50 smithi079 systemd[1]: Started Ceph mds.b for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-17T23:54:51.577 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:54:51.735 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:51 smithi079 ceph-mon[36241]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:51.735 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.735 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.735 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.735 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.735 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.735 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:51 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] up:boot 2024-03-17T23:54:51.735 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:51 smithi079 ceph-mon[36241]: fsmap 12 up:standby 2024-03-17T23:54:51.735 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-17T23:54:51.736 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-17T23:54:51.736 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:54:51.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.782+0000 7f9b15000640 1 -- 172.21.15.12:0/1418333925 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b10102280 msgr2=0x7f9b10104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:51.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.782+0000 7f9b15000640 1 --2- 172.21.15.12:0/1418333925 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b10102280 0x7f9b10104670 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9b00009900 tx=0x7f9b0002bc50 comp rx=0 tx=0).stop 2024-03-17T23:54:51.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.782+0000 7f9b15000640 1 -- 172.21.15.12:0/1418333925 shutdown_connections 2024-03-17T23:54:51.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.782+0000 7f9b15000640 1 --2- 172.21.15.12:0/1418333925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b101074e0 0x7f9b10109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:51.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.782+0000 7f9b15000640 1 --2- 172.21.15.12:0/1418333925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b10104bb0 0x7f9b10106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:51.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.782+0000 7f9b15000640 1 --2- 172.21.15.12:0/1418333925 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b10102280 0x7f9b10104670 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:51.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.782+0000 7f9b15000640 1 -- 172.21.15.12:0/1418333925 >> 172.21.15.12:0/1418333925 conn(0x7f9b100fbc70 msgr2=0x7f9b100fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:51.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.783+0000 7f9b15000640 1 -- 172.21.15.12:0/1418333925 shutdown_connections 2024-03-17T23:54:51.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.783+0000 7f9b15000640 1 -- 172.21.15.12:0/1418333925 wait complete. 2024-03-17T23:54:51.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.784+0000 7f9b15000640 1 Processor -- start 2024-03-17T23:54:51.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.784+0000 7f9b15000640 1 -- start start 2024-03-17T23:54:51.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.784+0000 7f9b15000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b10102280 0x7f9b10101bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:51.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.784+0000 7f9b15000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b10104bb0 0x7f9b10100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:51.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.784+0000 7f9b15000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b101074e0 0x7f9b10100770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.784+0000 7f9b15000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1010d4d0 con 0x7f9b10102280 2024-03-17T23:54:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.784+0000 7f9b15000640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1010d350 con 0x7f9b10104bb0 2024-03-17T23:54:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.784+0000 7f9b15000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1010d650 con 0x7f9b101074e0 2024-03-17T23:54:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.784+0000 7f9b0de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b10102280 0x7f9b10101bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.784+0000 7f9b0de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b10102280 0x7f9b10101bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52662/0 (socket says 172.21.15.12:52662) 2024-03-17T23:54:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.784+0000 7f9b0de00640 1 -- 172.21.15.12:0/2058793125 learned_addr learned my addr 172.21.15.12:0/2058793125 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.784+0000 7f9b0d400640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b10104bb0 0x7f9b10100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.785+0000 7f9b0e800640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b101074e0 0x7f9b10100770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.785+0000 7f9b0de00640 1 -- 172.21.15.12:0/2058793125 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b101074e0 msgr2=0x7f9b10100770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.785+0000 7f9b0de00640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b101074e0 0x7f9b10100770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.785+0000 7f9b0de00640 1 -- 172.21.15.12:0/2058793125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b10104bb0 msgr2=0x7f9b10100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:51.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.785+0000 7f9b0de00640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b10104bb0 0x7f9b10100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:51.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.785+0000 7f9b0de00640 1 -- 172.21.15.12:0/2058793125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b10101000 con 0x7f9b10102280 2024-03-17T23:54:51.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.785+0000 7f9b0de00640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b10102280 0x7f9b10101bf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9b0002c260 tx=0x7f9b0002c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:51.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.786+0000 7f9b06a00640 1 -- 172.21.15.12:0/2058793125 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b00041070 con 0x7f9b10102280 2024-03-17T23:54:51.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.786+0000 7f9b06a00640 1 -- 172.21.15.12:0/2058793125 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b00004450 con 0x7f9b10102280 2024-03-17T23:54:51.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.786+0000 7f9b06a00640 1 -- 172.21.15.12:0/2058793125 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b00034440 con 0x7f9b10102280 2024-03-17T23:54:51.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.786+0000 7f9b15000640 1 -- 172.21.15.12:0/2058793125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b101a1c30 con 0x7f9b10102280 2024-03-17T23:54:51.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.786+0000 7f9b15000640 1 -- 172.21.15.12:0/2058793125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b101a2130 con 0x7f9b10102280 2024-03-17T23:54:51.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.788+0000 7f9b15000640 1 -- 172.21.15.12:0/2058793125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b100fd760 con 0x7f9b10102280 2024-03-17T23:54:51.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.790+0000 7f9b06a00640 1 -- 172.21.15.12:0/2058793125 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9b00044070 con 0x7f9b10102280 2024-03-17T23:54:51.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.791+0000 7f9b06a00640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9af40a2800 0x7f9af40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:51.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.791+0000 7f9b06a00640 1 -- 172.21.15.12:0/2058793125 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9b000f0110 con 0x7f9b10102280 2024-03-17T23:54:51.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.791+0000 7f9b0d400640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9af40a2800 0x7f9af40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:51.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.792+0000 7f9b0d400640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9af40a2800 0x7f9af40a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9b10101960 tx=0x7f9afc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:51.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:51.795+0000 7f9b06a00640 1 -- 172.21.15.12:0/2058793125 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9b000bcf60 con 0x7f9b10102280 2024-03-17T23:54:51.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:51 smithi191 ceph-mon[36708]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:51.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:51 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] up:boot 2024-03-17T23:54:51.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:51 smithi191 ceph-mon[36708]: fsmap 12 up:standby 2024-03-17T23:54:51.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-17T23:54:51.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-17T23:54:51.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:54:51.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:51 smithi012 ceph-mon[31541]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:51.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:51.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:51 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] up:boot 2024-03-17T23:54:51.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:51 smithi012 ceph-mon[31541]: fsmap 12 up:standby 2024-03-17T23:54:51.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-17T23:54:51.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-17T23:54:51.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:54:52.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.048+0000 7f9b15000640 1 -- 172.21.15.12:0/2058793125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b1002d1b0 con 0x7f9b10102280 2024-03-17T23:54:52.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.049+0000 7f9b06a00640 1 -- 172.21.15.12:0/2058793125 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10482 (secure 0 0 0) 0x7f9b000c1000 con 0x7f9b10102280 2024-03-17T23:54:52.050 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:52.050 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/2073607248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":2073607248},{"type":"v1","addr":"172.21.15.12:6841","nonce":2073607248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/2478742711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":2478742711},{"type":"v1","addr":"172.21.15.79:6839","nonce":2478742711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-17T23:54:52.051 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 13 2024-03-17T23:54:52.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.052+0000 7f9b15000640 1 -- 172.21.15.12:0/2058793125 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9af40a2800 msgr2=0x7f9af40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:52.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.052+0000 7f9b15000640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9af40a2800 0x7f9af40a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9b10101960 tx=0x7f9afc005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:52.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.052+0000 7f9b15000640 1 -- 172.21.15.12:0/2058793125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b10102280 msgr2=0x7f9b10101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:52.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.052+0000 7f9b15000640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b10102280 0x7f9b10101bf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9b0002c260 tx=0x7f9b0002c670 comp rx=0 tx=0).stop 2024-03-17T23:54:52.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.052+0000 7f9b15000640 1 -- 172.21.15.12:0/2058793125 shutdown_connections 2024-03-17T23:54:52.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.052+0000 7f9b15000640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9af40a2800 0x7f9af40a4cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.052+0000 7f9b15000640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b101074e0 0x7f9b10100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.052+0000 7f9b15000640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b10104bb0 0x7f9b10100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.052+0000 7f9b15000640 1 --2- 172.21.15.12:0/2058793125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b10102280 0x7f9b10101bf0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.052+0000 7f9b15000640 1 -- 172.21.15.12:0/2058793125 >> 172.21.15.12:0/2058793125 conn(0x7f9b100fbc70 msgr2=0x7f9b10102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.052+0000 7f9b15000640 1 -- 172.21.15.12:0/2058793125 shutdown_connections 2024-03-17T23:54:52.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.052+0000 7f9b15000640 1 -- 172.21.15.12:0/2058793125 wait complete. 2024-03-17T23:54:52.074 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:54:52.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.299+0000 7f5d2a200640 1 -- 172.21.15.12:0/1569400741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d24077d80 msgr2=0x7f5d24078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:52.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.299+0000 7f5d2a200640 1 --2- 172.21.15.12:0/1569400741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d24077d80 0x7f5d24078620 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009ca0 tx=0x7f5d1c02c100 comp rx=0 tx=0).stop 2024-03-17T23:54:52.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.299+0000 7f5d2a200640 1 -- 172.21.15.12:0/1569400741 shutdown_connections 2024-03-17T23:54:52.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.300+0000 7f5d2a200640 1 --2- 172.21.15.12:0/1569400741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d24077d80 0x7f5d24078620 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.300+0000 7f5d2a200640 1 --2- 172.21.15.12:0/1569400741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5d240773e0 0x7f5d24077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.300+0000 7f5d2a200640 1 --2- 172.21.15.12:0/1569400741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d240761a0 0x7f5d240765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.300+0000 7f5d2a200640 1 -- 172.21.15.12:0/1569400741 >> 172.21.15.12:0/1569400741 conn(0x7f5d240fef90 msgr2=0x7f5d241013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:52.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.300+0000 7f5d2a200640 1 -- 172.21.15.12:0/1569400741 shutdown_connections 2024-03-17T23:54:52.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.300+0000 7f5d2a200640 1 -- 172.21.15.12:0/1569400741 wait complete. 2024-03-17T23:54:52.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.301+0000 7f5d2a200640 1 Processor -- start 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.301+0000 7f5d2a200640 1 -- start start 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.301+0000 7f5d2a200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d240761a0 0x7f5d2419e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.301+0000 7f5d2a200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d240773e0 0x7f5d2419e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.301+0000 7f5d2a200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5d24077d80 0x7f5d241a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.301+0000 7f5d2a200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5d24113a90 con 0x7f5d240773e0 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.301+0000 7f5d2a200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5d24113910 con 0x7f5d24077d80 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.301+0000 7f5d2a200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5d24113c10 con 0x7f5d240761a0 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.301+0000 7f5d23400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d240761a0 0x7f5d2419e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.301+0000 7f5d23400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d240761a0 0x7f5d2419e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:44684/0 (socket says 172.21.15.12:44684) 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.301+0000 7f5d23400640 1 -- 172.21.15.12:0/621939027 learned_addr learned my addr 172.21.15.12:0/621939027 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.301+0000 7f5d22a00640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d240773e0 0x7f5d2419e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.301+0000 7f5d23e00640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5d24077d80 0x7f5d241a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.302+0000 7f5d23400640 1 -- 172.21.15.12:0/621939027 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5d24077d80 msgr2=0x7f5d241a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:52.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.302+0000 7f5d23400640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5d24077d80 0x7f5d241a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.302+0000 7f5d23400640 1 -- 172.21.15.12:0/621939027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d240773e0 msgr2=0x7f5d2419e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:52.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.302+0000 7f5d23400640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d240773e0 0x7f5d2419e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.302+0000 7f5d23400640 1 -- 172.21.15.12:0/621939027 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d241a5c60 con 0x7f5d240761a0 2024-03-17T23:54:52.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.302+0000 7f5d23400640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d240761a0 0x7f5d2419e040 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400e9f0 tx=0x7f5d1400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:52.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.302+0000 7f5d1be00640 1 -- 172.21.15.12:0/621939027 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1400cd80 con 0x7f5d240761a0 2024-03-17T23:54:52.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.303+0000 7f5d1be00640 1 -- 172.21.15.12:0/621939027 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d14010910 con 0x7f5d240761a0 2024-03-17T23:54:52.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.303+0000 7f5d2a200640 1 -- 172.21.15.12:0/621939027 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d241a5f50 con 0x7f5d240761a0 2024-03-17T23:54:52.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.303+0000 7f5d1be00640 1 -- 172.21.15.12:0/621939027 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d14018990 con 0x7f5d240761a0 2024-03-17T23:54:52.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.303+0000 7f5d2a200640 1 -- 172.21.15.12:0/621939027 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d241a6410 con 0x7f5d240761a0 2024-03-17T23:54:52.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.304+0000 7f5d2a200640 1 -- 172.21.15.12:0/621939027 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d24110370 con 0x7f5d240761a0 2024-03-17T23:54:52.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.307+0000 7f5d1be00640 1 -- 172.21.15.12:0/621939027 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5d14010ac0 con 0x7f5d240761a0 2024-03-17T23:54:52.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.309+0000 7f5d1be00640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5d080a2880 0x7f5d080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:52.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.309+0000 7f5d1be00640 1 -- 172.21.15.12:0/621939027 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5d140ce910 con 0x7f5d240761a0 2024-03-17T23:54:52.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.309+0000 7f5d22a00640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5d080a2880 0x7f5d080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:52.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.310+0000 7f5d22a00640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5d080a2880 0x7f5d080a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5d100095f0 tx=0x7f5d10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:52.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.313+0000 7f5d1be00640 1 -- 172.21.15.12:0/621939027 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5d1409b760 con 0x7f5d240761a0 2024-03-17T23:54:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.545+0000 7f5d2a200640 1 -- 172.21.15.12:0/621939027 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d24062480 con 0x7f5d240761a0 2024-03-17T23:54:52.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.548+0000 7f5d1be00640 1 -- 172.21.15.12:0/621939027 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18372 (secure 0 0 0) 0x7f5d140bb020 con 0x7f5d240761a0 2024-03-17T23:54:52.549 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:52.549 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":81,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:53:58.251750+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:54:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.552+0000 7f5d2a200640 1 -- 172.21.15.12:0/621939027 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5d080a2880 msgr2=0x7f5d080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.552+0000 7f5d2a200640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5d080a2880 0x7f5d080a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5d100095f0 tx=0x7f5d10005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.552+0000 7f5d2a200640 1 -- 172.21.15.12:0/621939027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d240761a0 msgr2=0x7f5d2419e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.552+0000 7f5d2a200640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d240761a0 0x7f5d2419e040 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400e9f0 tx=0x7f5d1400eeb0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.552+0000 7f5d2a200640 1 -- 172.21.15.12:0/621939027 shutdown_connections 2024-03-17T23:54:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.552+0000 7f5d2a200640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5d080a2880 0x7f5d080a4d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.552+0000 7f5d2a200640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5d24077d80 0x7f5d241a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.552+0000 7f5d2a200640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d240773e0 0x7f5d2419e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.552+0000 7f5d2a200640 1 --2- 172.21.15.12:0/621939027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d240761a0 0x7f5d2419e040 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.552+0000 7f5d2a200640 1 -- 172.21.15.12:0/621939027 >> 172.21.15.12:0/621939027 conn(0x7f5d240fef90 msgr2=0x7f5d24100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.552+0000 7f5d2a200640 1 -- 172.21.15.12:0/621939027 shutdown_connections 2024-03-17T23:54:52.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.552+0000 7f5d2a200640 1 -- 172.21.15.12:0/621939027 wait complete. 2024-03-17T23:54:52.573 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:49:38.640282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:54:52.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:54:52.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:52 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2058793125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:52.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:52 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2058793125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.800+0000 7f856a800640 1 -- 172.21.15.12:0/3637033545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8564104ba0 msgr2=0x7f8564106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.800+0000 7f856a800640 1 --2- 172.21.15.12:0/3637033545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8564104ba0 0x7f8564106f90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8550009900 tx=0x7f855002bc30 comp rx=0 tx=0).stop 2024-03-17T23:54:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.800+0000 7f856a800640 1 -- 172.21.15.12:0/3637033545 shutdown_connections 2024-03-17T23:54:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.800+0000 7f856a800640 1 --2- 172.21.15.12:0/3637033545 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85641074d0 0x7f8564109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.800+0000 7f856a800640 1 --2- 172.21.15.12:0/3637033545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8564104ba0 0x7f8564106f90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.800+0000 7f856a800640 1 --2- 172.21.15.12:0/3637033545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8564102270 0x7f8564104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.800+0000 7f856a800640 1 -- 172.21.15.12:0/3637033545 >> 172.21.15.12:0/3637033545 conn(0x7f85640fbc80 msgr2=0x7f85640fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.800+0000 7f856a800640 1 -- 172.21.15.12:0/3637033545 shutdown_connections 2024-03-17T23:54:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.800+0000 7f856a800640 1 -- 172.21.15.12:0/3637033545 wait complete. 2024-03-17T23:54:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.801+0000 7f856a800640 1 Processor -- start 2024-03-17T23:54:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.801+0000 7f856a800640 1 -- start start 2024-03-17T23:54:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f856a800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8564102270 0x7f8564199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f856a800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8564104ba0 0x7f856419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f856a800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85641074d0 0x7f85641a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f856a800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f856410d460 con 0x7f8564102270 2024-03-17T23:54:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f856a800640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f856410d2e0 con 0x7f8564104ba0 2024-03-17T23:54:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f856a800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f856410d5e0 con 0x7f85641074d0 2024-03-17T23:54:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f8563400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8564102270 0x7f8564199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f8562a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8564104ba0 0x7f856419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f8562a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8564104ba0 0x7f856419a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:51114/0 (socket says 172.21.15.12:51114) 2024-03-17T23:54:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f8562a00640 1 -- 172.21.15.12:0/1200499845 learned_addr learned my addr 172.21.15.12:0/1200499845 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f8563e00640 1 --2- 172.21.15.12:0/1200499845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85641074d0 0x7f85641a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f8562a00640 1 -- 172.21.15.12:0/1200499845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85641074d0 msgr2=0x7f85641a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f8562a00640 1 --2- 172.21.15.12:0/1200499845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85641074d0 0x7f85641a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f8562a00640 1 -- 172.21.15.12:0/1200499845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8564102270 msgr2=0x7f8564199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.802+0000 7f8562a00640 1 --2- 172.21.15.12:0/1200499845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8564102270 0x7f8564199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.803+0000 7f8562a00640 1 -- 172.21.15.12:0/1200499845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85641a1970 con 0x7f8564104ba0 2024-03-17T23:54:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.803+0000 7f8562a00640 1 --2- 172.21.15.12:0/1200499845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8564104ba0 0x7f856419a290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8550000c00 tx=0x7f855002c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.803+0000 7f855be00640 1 -- 172.21.15.12:0/1200499845 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8550041070 con 0x7f8564104ba0 2024-03-17T23:54:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.803+0000 7f856a800640 1 -- 172.21.15.12:0/1200499845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85641a1c00 con 0x7f8564104ba0 2024-03-17T23:54:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.803+0000 7f855be00640 1 -- 172.21.15.12:0/1200499845 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8550004450 con 0x7f8564104ba0 2024-03-17T23:54:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.803+0000 7f855be00640 1 -- 172.21.15.12:0/1200499845 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8550003df0 con 0x7f8564104ba0 2024-03-17T23:54:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.804+0000 7f856a800640 1 -- 172.21.15.12:0/1200499845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85641a2140 con 0x7f8564104ba0 2024-03-17T23:54:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.805+0000 7f856a800640 1 -- 172.21.15.12:0/1200499845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8564109020 con 0x7f8564104ba0 2024-03-17T23:54:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.808+0000 7f855be00640 1 -- 172.21.15.12:0/1200499845 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8550044070 con 0x7f8564104ba0 2024-03-17T23:54:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.809+0000 7f855be00640 1 --2- 172.21.15.12:0/1200499845 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f85480a2800 0x7f85480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.810+0000 7f855be00640 1 -- 172.21.15.12:0/1200499845 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f85500f0060 con 0x7f8564104ba0 2024-03-17T23:54:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.810+0000 7f8563400640 1 --2- 172.21.15.12:0/1200499845 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f85480a2800 0x7f85480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.810+0000 7f8563400640 1 --2- 172.21.15.12:0/1200499845 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f85480a2800 0x7f85480a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8554009bd0 tx=0x7f8554005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:52.814+0000 7f855be00640 1 -- 172.21.15.12:0/1200499845 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f85500bd630 con 0x7f8564104ba0 2024-03-17T23:54:52.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:52 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2058793125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:54:53.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.030+0000 7f856a800640 1 -- 172.21.15.12:0/1200499845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f856402d380 con 0x7f8564104ba0 2024-03-17T23:54:53.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.030+0000 7f855be00640 1 -- 172.21.15.12:0/1200499845 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f85500bcf30 con 0x7f8564104ba0 2024-03-17T23:54:53.031 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-17T23:54:53.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.033+0000 7f856a800640 1 -- 172.21.15.12:0/1200499845 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f85480a2800 msgr2=0x7f85480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:53.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.033+0000 7f856a800640 1 --2- 172.21.15.12:0/1200499845 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f85480a2800 0x7f85480a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8554009bd0 tx=0x7f8554005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:53.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.033+0000 7f856a800640 1 -- 172.21.15.12:0/1200499845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8564104ba0 msgr2=0x7f856419a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:53.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.033+0000 7f856a800640 1 --2- 172.21.15.12:0/1200499845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8564104ba0 0x7f856419a290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8550000c00 tx=0x7f855002c640 comp rx=0 tx=0).stop 2024-03-17T23:54:53.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.033+0000 7f856a800640 1 -- 172.21.15.12:0/1200499845 shutdown_connections 2024-03-17T23:54:53.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.033+0000 7f856a800640 1 --2- 172.21.15.12:0/1200499845 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f85480a2800 0x7f85480a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.033+0000 7f856a800640 1 --2- 172.21.15.12:0/1200499845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85641074d0 0x7f85641a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.033+0000 7f856a800640 1 --2- 172.21.15.12:0/1200499845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8564104ba0 0x7f856419a290 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.033+0000 7f856a800640 1 --2- 172.21.15.12:0/1200499845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8564102270 0x7f8564199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.033+0000 7f856a800640 1 -- 172.21.15.12:0/1200499845 >> 172.21.15.12:0/1200499845 conn(0x7f85640fbc80 msgr2=0x7f8564101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:53.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.033+0000 7f856a800640 1 -- 172.21.15.12:0/1200499845 shutdown_connections 2024-03-17T23:54:53.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.033+0000 7f856a800640 1 -- 172.21.15.12:0/1200499845 wait complete. 2024-03-17T23:54:53.056 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-17T23:54:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.261+0000 7fa49f200640 1 -- 172.21.15.12:0/3848719134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa498102830 msgr2=0x7fa498102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.261+0000 7fa49f200640 1 --2- 172.21.15.12:0/3848719134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa498102830 0x7fa498102cb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa488009900 tx=0x7fa48802bc30 comp rx=0 tx=0).stop 2024-03-17T23:54:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.261+0000 7fa49f200640 1 -- 172.21.15.12:0/3848719134 shutdown_connections 2024-03-17T23:54:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.261+0000 7fa49f200640 1 --2- 172.21.15.12:0/3848719134 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4981031f0 0x7fa498109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.261+0000 7fa49f200640 1 --2- 172.21.15.12:0/3848719134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa498102830 0x7fa498102cb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.261+0000 7fa49f200640 1 --2- 172.21.15.12:0/3848719134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa498101640 0x7fa498101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.261+0000 7fa49f200640 1 -- 172.21.15.12:0/3848719134 >> 172.21.15.12:0/3848719134 conn(0x7fa4980fcbb0 msgr2=0x7fa4980feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:53.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.261+0000 7fa49f200640 1 -- 172.21.15.12:0/3848719134 shutdown_connections 2024-03-17T23:54:53.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.261+0000 7fa49f200640 1 -- 172.21.15.12:0/3848719134 wait complete. 2024-03-17T23:54:53.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.263+0000 7fa49f200640 1 Processor -- start 2024-03-17T23:54:53.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.263+0000 7fa49f200640 1 -- start start 2024-03-17T23:54:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.263+0000 7fa49f200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa498101640 0x7fa498075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.263+0000 7fa49f200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa498102830 0x7fa498073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.263+0000 7fa49f200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4981031f0 0x7fa498074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.263+0000 7fa49f200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa49810d310 con 0x7fa498101640 2024-03-17T23:54:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.263+0000 7fa49f200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa49810d190 con 0x7fa4981031f0 2024-03-17T23:54:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.263+0000 7fa49f200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa49810d490 con 0x7fa498102830 2024-03-17T23:54:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.264+0000 7fa49de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa498101640 0x7fa498075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.264+0000 7fa49de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa498101640 0x7fa498075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52728/0 (socket says 172.21.15.12:52728) 2024-03-17T23:54:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.264+0000 7fa49de00640 1 -- 172.21.15.12:0/2088644141 learned_addr learned my addr 172.21.15.12:0/2088644141 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.264+0000 7fa49e800640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4981031f0 0x7fa498074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.264+0000 7fa49d400640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa498102830 0x7fa498073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:53.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.264+0000 7fa49e800640 1 -- 172.21.15.12:0/2088644141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa498102830 msgr2=0x7fa498073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:53.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.264+0000 7fa49e800640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa498102830 0x7fa498073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.264+0000 7fa49e800640 1 -- 172.21.15.12:0/2088644141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa498101640 msgr2=0x7fa498075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:53.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.264+0000 7fa49e800640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa498101640 0x7fa498075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.264+0000 7fa49e800640 1 -- 172.21.15.12:0/2088644141 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4980749a0 con 0x7fa4981031f0 2024-03-17T23:54:53.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.265+0000 7fa49e800640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4981031f0 0x7fa498074110 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa49400ec50 tx=0x7fa49400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:53.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.265+0000 7fa492a00640 1 -- 172.21.15.12:0/2088644141 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa494003b70 con 0x7fa4981031f0 2024-03-17T23:54:53.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.265+0000 7fa492a00640 1 -- 172.21.15.12:0/2088644141 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa494003d10 con 0x7fa4981031f0 2024-03-17T23:54:53.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.265+0000 7fa49f200640 1 -- 172.21.15.12:0/2088644141 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4981a1a80 con 0x7fa4981031f0 2024-03-17T23:54:53.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.265+0000 7fa492a00640 1 -- 172.21.15.12:0/2088644141 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa494010890 con 0x7fa4981031f0 2024-03-17T23:54:53.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.265+0000 7fa49f200640 1 -- 172.21.15.12:0/2088644141 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4981a1f50 con 0x7fa4981031f0 2024-03-17T23:54:53.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.266+0000 7fa49f200640 1 -- 172.21.15.12:0/2088644141 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4980684f0 con 0x7fa4981031f0 2024-03-17T23:54:53.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.268+0000 7fa492a00640 1 -- 172.21.15.12:0/2088644141 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa494003fc0 con 0x7fa4981031f0 2024-03-17T23:54:53.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.269+0000 7fa492a00640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4800a2800 0x7fa4800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:53.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.270+0000 7fa492a00640 1 -- 172.21.15.12:0/2088644141 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa4940cec50 con 0x7fa4981031f0 2024-03-17T23:54:53.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.270+0000 7fa49de00640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4800a2800 0x7fa4800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:53.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.270+0000 7fa49de00640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4800a2800 0x7fa4800a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0058c0 tx=0x7fa48c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:53.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.274+0000 7fa492a00640 1 -- 172.21.15.12:0/2088644141 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa49409c0f0 con 0x7fa4981031f0 2024-03-17T23:54:53.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.511+0000 7fa49f200640 1 -- 172.21.15.12:0/2088644141 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa498062410 con 0x7fa4981031f0 2024-03-17T23:54:53.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.512+0000 7fa492a00640 1 -- 172.21.15.12:0/2088644141 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fa49409b9f0 con 0x7fa4981031f0 2024-03-17T23:54:53.512 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:54:53.512 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-03-17T23:54:53.512 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-17T23:54:53.512 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-03-17T23:54:53.513 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-03-17T23:54:53.513 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-17T23:54:53.513 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-03-17T23:54:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.514+0000 7fa49f200640 1 -- 172.21.15.12:0/2088644141 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4800a2800 msgr2=0x7fa4800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.514+0000 7fa49f200640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4800a2800 0x7fa4800a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0058c0 tx=0x7fa48c005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.514+0000 7fa49f200640 1 -- 172.21.15.12:0/2088644141 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4981031f0 msgr2=0x7fa498074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.514+0000 7fa49f200640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4981031f0 0x7fa498074110 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa49400ec50 tx=0x7fa49400c6a0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.515+0000 7fa49f200640 1 -- 172.21.15.12:0/2088644141 shutdown_connections 2024-03-17T23:54:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.515+0000 7fa49f200640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4800a2800 0x7fa4800a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.515+0000 7fa49f200640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4981031f0 0x7fa498074110 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.515+0000 7fa49f200640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa498102830 0x7fa498073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.515+0000 7fa49f200640 1 --2- 172.21.15.12:0/2088644141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa498101640 0x7fa498075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.515+0000 7fa49f200640 1 -- 172.21.15.12:0/2088644141 >> 172.21.15.12:0/2088644141 conn(0x7fa4980fcbb0 msgr2=0x7fa498105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.515+0000 7fa49f200640 1 -- 172.21.15.12:0/2088644141 shutdown_connections 2024-03-17T23:54:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.515+0000 7fa49f200640 1 -- 172.21.15.12:0/2088644141 wait complete. 2024-03-17T23:54:53.533 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-17T23:54:53.534 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-17T23:54:53.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.750+0000 7f5947e00640 1 -- 172.21.15.12:0/3379107557 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59480fffb0 msgr2=0x7f59481003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:53.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.750+0000 7f5947e00640 1 --2- 172.21.15.12:0/3379107557 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59480fffb0 0x7f59481003d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5938009900 tx=0x7f593802bcd0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.750+0000 7f5947e00640 1 -- 172.21.15.12:0/3379107557 shutdown_connections 2024-03-17T23:54:53.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.750+0000 7f5947e00640 1 --2- 172.21.15.12:0/3379107557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59481012b0 0x7f5948101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.750+0000 7f5947e00640 1 --2- 172.21.15.12:0/3379107557 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5948100910 0x7f5948100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.750+0000 7f5947e00640 1 --2- 172.21.15.12:0/3379107557 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59480fffb0 0x7f59481003d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.750+0000 7f5947e00640 1 -- 172.21.15.12:0/3379107557 >> 172.21.15.12:0/3379107557 conn(0x7f59480fbb50 msgr2=0x7f59480fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:53.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.750+0000 7f5947e00640 1 -- 172.21.15.12:0/3379107557 shutdown_connections 2024-03-17T23:54:53.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.750+0000 7f5947e00640 1 -- 172.21.15.12:0/3379107557 wait complete. 2024-03-17T23:54:53.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.751+0000 7f5947e00640 1 Processor -- start 2024-03-17T23:54:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.752+0000 7f5947e00640 1 -- start start 2024-03-17T23:54:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.752+0000 7f5947e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59480fffb0 0x7f5948199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.752+0000 7f5947e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5948100910 0x7f594819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.752+0000 7f5947e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59481012b0 0x7f59481a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.752+0000 7f5947e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f594810f590 con 0x7f5948100910 2024-03-17T23:54:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.752+0000 7f5947e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f594810f410 con 0x7f59481012b0 2024-03-17T23:54:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.752+0000 7f5947e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f594810f710 con 0x7f59480fffb0 2024-03-17T23:54:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.752+0000 7f5946000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5948100910 0x7f594819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.752+0000 7f5947400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59481012b0 0x7f59481a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.752+0000 7f5946000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5948100910 0x7f594819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52736/0 (socket says 172.21.15.12:52736) 2024-03-17T23:54:53.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.752+0000 7f5946000640 1 -- 172.21.15.12:0/713186795 learned_addr learned my addr 172.21.15.12:0/713186795 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:53.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.752+0000 7f5946a00640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59480fffb0 0x7f5948199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:53.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.753+0000 7f5946000640 1 -- 172.21.15.12:0/713186795 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59480fffb0 msgr2=0x7f5948199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:53.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.753+0000 7f5946000640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59480fffb0 0x7f5948199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.753+0000 7f5946000640 1 -- 172.21.15.12:0/713186795 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59481012b0 msgr2=0x7f59481a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:53.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.753+0000 7f5946000640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59481012b0 0x7f59481a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:53.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.753+0000 7f5946000640 1 -- 172.21.15.12:0/713186795 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59481a1800 con 0x7f5948100910 2024-03-17T23:54:53.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.753+0000 7f5947400640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59481012b0 0x7f59481a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:53.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.753+0000 7f5946000640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5948100910 0x7f594819a120 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f593400e9f0 tx=0x7f593400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:53.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.756+0000 7f593f400640 1 -- 172.21.15.12:0/713186795 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593400cd80 con 0x7f5948100910 2024-03-17T23:54:53.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.756+0000 7f5947e00640 1 -- 172.21.15.12:0/713186795 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59481a1af0 con 0x7f5948100910 2024-03-17T23:54:53.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.756+0000 7f593f400640 1 -- 172.21.15.12:0/713186795 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5934005500 con 0x7f5948100910 2024-03-17T23:54:53.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.756+0000 7f593f400640 1 -- 172.21.15.12:0/713186795 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5934010690 con 0x7f5948100910 2024-03-17T23:54:53.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.756+0000 7f5947e00640 1 -- 172.21.15.12:0/713186795 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59481a2040 con 0x7f5948100910 2024-03-17T23:54:53.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.757+0000 7f5947e00640 1 -- 172.21.15.12:0/713186795 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f594810ad00 con 0x7f5948100910 2024-03-17T23:54:53.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.760+0000 7f593f400640 1 -- 172.21.15.12:0/713186795 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f59340046c0 con 0x7f5948100910 2024-03-17T23:54:53.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.761+0000 7f593f400640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f592c0a2880 0x7f592c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:53.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.762+0000 7f593f400640 1 -- 172.21.15.12:0/713186795 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f59340ce9c0 con 0x7f5948100910 2024-03-17T23:54:53.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.762+0000 7f5946a00640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f592c0a2880 0x7f592c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:53.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.762+0000 7f5946a00640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f592c0a2880 0x7f592c0a4d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f593802c2e0 tx=0x7f593802c920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:53.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.766+0000 7f593f400640 1 -- 172.21.15.12:0/713186795 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f593409be60 con 0x7f5948100910 2024-03-17T23:54:53.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:53 smithi079 ceph-mon[36241]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:53.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:53 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:53.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:53 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:53.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:53 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/621939027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:54:53.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:53 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:53.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:53 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:53.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:53 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1200499845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:54:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:53 smithi191 ceph-mon[36708]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:53 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:53 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:53 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/621939027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:54:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:53 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:53 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:53 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1200499845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:54:53.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:53 smithi012 ceph-mon[31541]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:53.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:53 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:53.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:53 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:53.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:53 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/621939027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:54:53.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:53 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:53.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:53 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:53.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:53 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1200499845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:54:53.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:53.991+0000 7f5947e00640 1 -- 172.21.15.12:0/713186795 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f59480623e0 con 0x7f5948100910 2024-03-17T23:54:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:54.769+0000 7f593f400640 1 -- 172.21.15.12:0/713186795 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f593409b760 con 0x7f5948100910 2024-03-17T23:54:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-03-17T23:54:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:54.774+0000 7f5947e00640 1 -- 172.21.15.12:0/713186795 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f592c0a2880 msgr2=0x7f592c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:54.774+0000 7f5947e00640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f592c0a2880 0x7f592c0a4d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f593802c2e0 tx=0x7f593802c920 comp rx=0 tx=0).stop 2024-03-17T23:54:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:54.774+0000 7f5947e00640 1 -- 172.21.15.12:0/713186795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5948100910 msgr2=0x7f594819a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:54.774+0000 7f5947e00640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5948100910 0x7f594819a120 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f593400e9f0 tx=0x7f593400edf0 comp rx=0 tx=0).stop 2024-03-17T23:54:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:54.775+0000 7f5947e00640 1 -- 172.21.15.12:0/713186795 shutdown_connections 2024-03-17T23:54:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:54.775+0000 7f5947e00640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f592c0a2880 0x7f592c0a4d40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:54.775+0000 7f5947e00640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59481012b0 0x7f59481a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:54.775+0000 7f5947e00640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5948100910 0x7f594819a120 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:54.775+0000 7f5947e00640 1 --2- 172.21.15.12:0/713186795 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59480fffb0 0x7f5948199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:54.775+0000 7f5947e00640 1 -- 172.21.15.12:0/713186795 >> 172.21.15.12:0/713186795 conn(0x7f59480fbb50 msgr2=0x7f59480fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:54.776+0000 7f5947e00640 1 -- 172.21.15.12:0/713186795 shutdown_connections 2024-03-17T23:54:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:54.776+0000 7f5947e00640 1 -- 172.21.15.12:0/713186795 wait complete. 2024-03-17T23:54:54.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:54 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2088644141' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-17T23:54:54.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:54.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:54.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:54.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:54:54.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:54.799 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:54 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2088644141' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-17T23:54:54.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:54.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:54.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:54.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:54:54.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:54.802 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-17T23:54:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:54 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2088644141' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-17T23:54:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:54:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:55.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.009+0000 7f14f8800640 1 -- 172.21.15.12:0/2205929847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14f00773e0 msgr2=0x7f14f0077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:55.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.009+0000 7f14f8800640 1 --2- 172.21.15.12:0/2205929847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14f00773e0 0x7f14f0077840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f14e0009900 tx=0x7f14e002bc30 comp rx=0 tx=0).stop 2024-03-17T23:54:55.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.009+0000 7f14f8800640 1 -- 172.21.15.12:0/2205929847 shutdown_connections 2024-03-17T23:54:55.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.009+0000 7f14f8800640 1 --2- 172.21.15.12:0/2205929847 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14f0077d80 0x7f14f0078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:55.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.009+0000 7f14f8800640 1 --2- 172.21.15.12:0/2205929847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14f00773e0 0x7f14f0077840 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:55.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.009+0000 7f14f8800640 1 --2- 172.21.15.12:0/2205929847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14f00761a0 0x7f14f00765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:55.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.009+0000 7f14f8800640 1 -- 172.21.15.12:0/2205929847 >> 172.21.15.12:0/2205929847 conn(0x7f14f00fef90 msgr2=0x7f14f01013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:55.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.009+0000 7f14f8800640 1 -- 172.21.15.12:0/2205929847 shutdown_connections 2024-03-17T23:54:55.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.009+0000 7f14f8800640 1 -- 172.21.15.12:0/2205929847 wait complete. 2024-03-17T23:54:55.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.010+0000 7f14f8800640 1 Processor -- start 2024-03-17T23:54:55.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.010+0000 7f14f8800640 1 -- start start 2024-03-17T23:54:55.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.010+0000 7f14f8800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14f00761a0 0x7f14f019e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f8800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14f00773e0 0x7f14f019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f8800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14f0077d80 0x7f14f01a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f8800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f14f0113af0 con 0x7f14f00773e0 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f8800640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f14f0113970 con 0x7f14f00761a0 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f8800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f14f0113c70 con 0x7f14f0077d80 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f5e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14f00761a0 0x7f14f019e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f5e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14f00761a0 0x7f14f019e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:49558/0 (socket says 172.21.15.12:49558) 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f5e00640 1 -- 172.21.15.12:0/437229481 learned_addr learned my addr 172.21.15.12:0/437229481 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f5400640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14f00773e0 0x7f14f019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f6800640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14f0077d80 0x7f14f01a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f5e00640 1 -- 172.21.15.12:0/437229481 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14f0077d80 msgr2=0x7f14f01a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f5e00640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14f0077d80 0x7f14f01a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f5e00640 1 -- 172.21.15.12:0/437229481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14f00773e0 msgr2=0x7f14f019e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f5e00640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14f00773e0 0x7f14f019e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.011+0000 7f14f5e00640 1 -- 172.21.15.12:0/437229481 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f01a5cf0 con 0x7f14f00761a0 2024-03-17T23:54:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.012+0000 7f14f5e00640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14f00761a0 0x7f14f019e0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00ebc0 tx=0x7f14ec00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.012+0000 7f14e6a00640 1 -- 172.21.15.12:0/437229481 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec010630 con 0x7f14f00761a0 2024-03-17T23:54:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.012+0000 7f14e6a00640 1 -- 172.21.15.12:0/437229481 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14ec010cb0 con 0x7f14f00761a0 2024-03-17T23:54:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.012+0000 7f14e6a00640 1 -- 172.21.15.12:0/437229481 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec018bd0 con 0x7f14f00761a0 2024-03-17T23:54:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.012+0000 7f14f8800640 1 -- 172.21.15.12:0/437229481 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14f01a5fe0 con 0x7f14f00761a0 2024-03-17T23:54:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.012+0000 7f14f8800640 1 -- 172.21.15.12:0/437229481 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14f01a6450 con 0x7f14f00761a0 2024-03-17T23:54:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.014+0000 7f14f8800640 1 -- 172.21.15.12:0/437229481 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14f0110390 con 0x7f14f00761a0 2024-03-17T23:54:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.015+0000 7f14e6a00640 1 -- 172.21.15.12:0/437229481 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f14ec0107d0 con 0x7f14f00761a0 2024-03-17T23:54:55.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.017+0000 7f14e6a00640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14dc0a2880 0x7f14dc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:55.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.017+0000 7f14e6a00640 1 -- 172.21.15.12:0/437229481 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f14ec0ced30 con 0x7f14f00761a0 2024-03-17T23:54:55.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.017+0000 7f14f5400640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14dc0a2880 0x7f14dc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.018+0000 7f14f5400640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14dc0a2880 0x7f14dc0a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f14e002c240 tx=0x7f14e002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.021+0000 7f14e6a00640 1 -- 172.21.15.12:0/437229481 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f14ec09ba10 con 0x7f14f00761a0 2024-03-17T23:54:55.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.227+0000 7f14f8800640 1 -- 172.21.15.12:0/437229481 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f14f0062480 con 0x7f14f00761a0 2024-03-17T23:54:55.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.830+0000 7f14e6a00640 1 -- 172.21.15.12:0/437229481 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7f14ec09fab0 con 0x7f14f00761a0 2024-03-17T23:54:55.832 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-03-17T23:54:55.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.838+0000 7f14f8800640 1 -- 172.21.15.12:0/437229481 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14dc0a2880 msgr2=0x7f14dc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:55.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.838+0000 7f14f8800640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14dc0a2880 0x7f14dc0a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f14e002c240 tx=0x7f14e002df90 comp rx=0 tx=0).stop 2024-03-17T23:54:55.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.838+0000 7f14f8800640 1 -- 172.21.15.12:0/437229481 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14f00761a0 msgr2=0x7f14f019e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:55.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.838+0000 7f14f8800640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14f00761a0 0x7f14f019e0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00ebc0 tx=0x7f14ec00c6a0 comp rx=0 tx=0).stop 2024-03-17T23:54:55.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.842+0000 7f14f8800640 1 -- 172.21.15.12:0/437229481 shutdown_connections 2024-03-17T23:54:55.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.842+0000 7f14f8800640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14dc0a2880 0x7f14dc0a4d40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:55.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.842+0000 7f14f8800640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14f0077d80 0x7f14f01a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:55.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.842+0000 7f14f8800640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14f00773e0 0x7f14f019e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:55.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.842+0000 7f14f8800640 1 --2- 172.21.15.12:0/437229481 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14f00761a0 0x7f14f019e0d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:55.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.842+0000 7f14f8800640 1 -- 172.21.15.12:0/437229481 >> 172.21.15.12:0/437229481 conn(0x7f14f00fef90 msgr2=0x7f14f0100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:55.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.842+0000 7f14f8800640 1 -- 172.21.15.12:0/437229481 shutdown_connections 2024-03-17T23:54:55.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:55.843+0000 7f14f8800640 1 -- 172.21.15.12:0/437229481 wait complete. 2024-03-17T23:54:55.875 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-17T23:54:56.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:55 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/713186795' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-17T23:54:56.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:55 smithi079 ceph-mon[36241]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:56.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:54:56.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:56.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:55 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/713186795' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-17T23:54:56.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:55 smithi079 ceph-mon[36241]: osdmap e82: 12 total, 12 up, 12 in 2024-03-17T23:54:56.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.055+0000 7ff577200640 1 -- 172.21.15.12:0/320224911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff578103290 msgr2=0x7ff578109b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:56.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.055+0000 7ff577200640 1 --2- 172.21.15.12:0/320224911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff578103290 0x7ff578109b20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff570009ca0 tx=0x7ff57002c0f0 comp rx=0 tx=0).stop 2024-03-17T23:54:56.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.055+0000 7ff577200640 1 -- 172.21.15.12:0/320224911 shutdown_connections 2024-03-17T23:54:56.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.055+0000 7ff577200640 1 --2- 172.21.15.12:0/320224911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff578103290 0x7ff578109b20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:56.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.055+0000 7ff577200640 1 --2- 172.21.15.12:0/320224911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5781028d0 0x7ff578102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:56.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.055+0000 7ff577200640 1 --2- 172.21.15.12:0/320224911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff5781016e0 0x7ff578101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:56.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.055+0000 7ff577200640 1 -- 172.21.15.12:0/320224911 >> 172.21.15.12:0/320224911 conn(0x7ff5780fcc50 msgr2=0x7ff5780ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:56.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.055+0000 7ff577200640 1 -- 172.21.15.12:0/320224911 shutdown_connections 2024-03-17T23:54:56.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.055+0000 7ff577200640 1 -- 172.21.15.12:0/320224911 wait complete. 2024-03-17T23:54:56.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.056+0000 7ff577200640 1 Processor -- start 2024-03-17T23:54:56.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.056+0000 7ff577200640 1 -- start start 2024-03-17T23:54:56.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff577200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff5781016e0 0x7ff578199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:56.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff577200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5781028d0 0x7ff57819a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:56.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff577200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff578103290 0x7ff5781a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:56.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff577200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff57810d340 con 0x7ff5781028d0 2024-03-17T23:54:56.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff577200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff57810d1c0 con 0x7ff5781016e0 2024-03-17T23:54:56.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff577200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff57810d4c0 con 0x7ff578103290 2024-03-17T23:54:56.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff575e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff5781016e0 0x7ff578199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:56.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff575e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff5781016e0 0x7ff578199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:49576/0 (socket says 172.21.15.12:49576) 2024-03-17T23:54:56.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff575e00640 1 -- 172.21.15.12:0/2637457076 learned_addr learned my addr 172.21.15.12:0/2637457076 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:56.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff576800640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff578103290 0x7ff5781a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:56.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff575400640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5781028d0 0x7ff57819a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:56.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff575e00640 1 -- 172.21.15.12:0/2637457076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff578103290 msgr2=0x7ff5781a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:56.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff575e00640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff578103290 0x7ff5781a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:56.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff575e00640 1 -- 172.21.15.12:0/2637457076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5781028d0 msgr2=0x7ff57819a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:56.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff575e00640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5781028d0 0x7ff57819a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:56.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.057+0000 7ff575e00640 1 -- 172.21.15.12:0/2637457076 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5781a1780 con 0x7ff5781016e0 2024-03-17T23:54:56.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.058+0000 7ff575e00640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff5781016e0 0x7ff578199b60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff56800e9c0 tx=0x7ff56800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:56.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.058+0000 7ff56ea00640 1 -- 172.21.15.12:0/2637457076 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56800cd80 con 0x7ff5781016e0 2024-03-17T23:54:56.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.058+0000 7ff56ea00640 1 -- 172.21.15.12:0/2637457076 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5680035f0 con 0x7ff5781016e0 2024-03-17T23:54:56.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.058+0000 7ff577200640 1 -- 172.21.15.12:0/2637457076 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5781a1a70 con 0x7ff5781016e0 2024-03-17T23:54:56.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.058+0000 7ff56ea00640 1 -- 172.21.15.12:0/2637457076 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568010690 con 0x7ff5781016e0 2024-03-17T23:54:56.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.058+0000 7ff577200640 1 -- 172.21.15.12:0/2637457076 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5781a2010 con 0x7ff5781016e0 2024-03-17T23:54:56.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.060+0000 7ff577200640 1 -- 172.21.15.12:0/2637457076 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5781a2240 con 0x7ff5781016e0 2024-03-17T23:54:56.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.061+0000 7ff56ea00640 1 -- 172.21.15.12:0/2637457076 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff5680037a0 con 0x7ff5781016e0 2024-03-17T23:54:56.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.063+0000 7ff56ea00640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff55c0a2880 0x7ff55c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:56.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.063+0000 7ff56ea00640 1 -- 172.21.15.12:0/2637457076 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7ff5680cec80 con 0x7ff5781016e0 2024-03-17T23:54:56.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.063+0000 7ff575400640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff55c0a2880 0x7ff55c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:56.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.063+0000 7ff575400640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff55c0a2880 0x7ff55c0a4d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff57819b070 tx=0x7ff564005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:56.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.068+0000 7ff56ea00640 1 -- 172.21.15.12:0/2637457076 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff56809be30 con 0x7ff5781016e0 2024-03-17T23:54:56.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:55 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/713186795' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-17T23:54:56.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:55 smithi191 ceph-mon[36708]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:56.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:54:56.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:56.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:55 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/713186795' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-17T23:54:56.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:55 smithi191 ceph-mon[36708]: osdmap e82: 12 total, 12 up, 12 in 2024-03-17T23:54:56.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:55 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/713186795' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-17T23:54:56.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:55 smithi012 ceph-mon[31541]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:56.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:54:56.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:56.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:55 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/713186795' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-17T23:54:56.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:55 smithi012 ceph-mon[31541]: osdmap e82: 12 total, 12 up, 12 in 2024-03-17T23:54:56.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:56.304+0000 7ff577200640 1 -- 172.21.15.12:0/2637457076 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff5781a26a0 con 0x7ff5781016e0 2024-03-17T23:54:57.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:56 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/437229481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-17T23:54:57.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:56 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-17T23:54:57.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:56 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-17T23:54:57.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:56 smithi191 ceph-mon[36708]: osdmap e83: 12 total, 12 up, 12 in 2024-03-17T23:54:57.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:56 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:57.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:56 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2637457076' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-17T23:54:57.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:56 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:57.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:56 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-17T23:54:57.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:56 smithi191 ceph-mon[36708]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-17T23:54:57.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:56 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:57.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:56 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:57.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:56 smithi191 ceph-mon[36708]: osdmap e84: 12 total, 12 up, 12 in 2024-03-17T23:54:57.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:56 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/437229481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-17T23:54:57.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:56 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-17T23:54:57.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:56 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-17T23:54:57.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:56 smithi012 ceph-mon[31541]: osdmap e83: 12 total, 12 up, 12 in 2024-03-17T23:54:57.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:56 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:57.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:56 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2637457076' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-17T23:54:57.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:56 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:57.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:56 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-17T23:54:57.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:56 smithi012 ceph-mon[31541]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-17T23:54:57.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:56 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:57.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:56 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:57.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:56 smithi012 ceph-mon[31541]: osdmap e84: 12 total, 12 up, 12 in 2024-03-17T23:54:57.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:56 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/437229481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-17T23:54:57.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:56 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-17T23:54:57.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:56 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-17T23:54:57.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:56 smithi079 ceph-mon[36241]: osdmap e83: 12 total, 12 up, 12 in 2024-03-17T23:54:57.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:56 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:57.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:56 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2637457076' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-17T23:54:57.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:56 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:57.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:56 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-17T23:54:57.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:56 smithi079 ceph-mon[36241]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-17T23:54:57.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:56 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:57.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:56 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:57.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:56 smithi079 ceph-mon[36241]: osdmap e84: 12 total, 12 up, 12 in 2024-03-17T23:54:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.312+0000 7ff56ea00640 1 -- 172.21.15.12:0/2637457076 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7ff56809b730 con 0x7ff5781016e0 2024-03-17T23:54:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 2 and data pool 3 2024-03-17T23:54:57.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.317+0000 7ff577200640 1 -- 172.21.15.12:0/2637457076 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff55c0a2880 msgr2=0x7ff55c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:57.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.317+0000 7ff577200640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff55c0a2880 0x7ff55c0a4d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff57819b070 tx=0x7ff564005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:57.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.317+0000 7ff577200640 1 -- 172.21.15.12:0/2637457076 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff5781016e0 msgr2=0x7ff578199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:57.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.317+0000 7ff577200640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff5781016e0 0x7ff578199b60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff56800e9c0 tx=0x7ff56800ee80 comp rx=0 tx=0).stop 2024-03-17T23:54:57.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.318+0000 7ff577200640 1 -- 172.21.15.12:0/2637457076 shutdown_connections 2024-03-17T23:54:57.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.318+0000 7ff577200640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff55c0a2880 0x7ff55c0a4d40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:57.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.318+0000 7ff577200640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff578103290 0x7ff5781a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:57.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.318+0000 7ff577200640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5781028d0 0x7ff57819a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:57.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.318+0000 7ff577200640 1 --2- 172.21.15.12:0/2637457076 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff5781016e0 0x7ff578199b60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:57.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.318+0000 7ff577200640 1 -- 172.21.15.12:0/2637457076 >> 172.21.15.12:0/2637457076 conn(0x7ff5780fcc50 msgr2=0x7ff578073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:57.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.319+0000 7ff577200640 1 -- 172.21.15.12:0/2637457076 shutdown_connections 2024-03-17T23:54:57.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.320+0000 7ff577200640 1 -- 172.21.15.12:0/2637457076 wait complete. 2024-03-17T23:54:57.342 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-17T23:54:57.343 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-17T23:54:57.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.543+0000 7fba40a00640 1 -- 172.21.15.12:0/3876795388 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba3c103270 msgr2=0x7fba3c109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:57.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.543+0000 7fba40a00640 1 --2- 172.21.15.12:0/3876795388 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba3c103270 0x7fba3c109b00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fba34009ca0 tx=0x7fba3402c100 comp rx=0 tx=0).stop 2024-03-17T23:54:57.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.543+0000 7fba40a00640 1 -- 172.21.15.12:0/3876795388 shutdown_connections 2024-03-17T23:54:57.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.543+0000 7fba40a00640 1 --2- 172.21.15.12:0/3876795388 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba3c103270 0x7fba3c109b00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:57.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.543+0000 7fba40a00640 1 --2- 172.21.15.12:0/3876795388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba3c1028b0 0x7fba3c102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:57.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.543+0000 7fba40a00640 1 --2- 172.21.15.12:0/3876795388 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c1016c0 0x7fba3c101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:57.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.543+0000 7fba40a00640 1 -- 172.21.15.12:0/3876795388 >> 172.21.15.12:0/3876795388 conn(0x7fba3c0fcc50 msgr2=0x7fba3c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:57.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.543+0000 7fba40a00640 1 -- 172.21.15.12:0/3876795388 shutdown_connections 2024-03-17T23:54:57.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.543+0000 7fba40a00640 1 -- 172.21.15.12:0/3876795388 wait complete. 2024-03-17T23:54:57.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.544+0000 7fba40a00640 1 Processor -- start 2024-03-17T23:54:57.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.544+0000 7fba40a00640 1 -- start start 2024-03-17T23:54:57.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.544+0000 7fba40a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c1016c0 0x7fba3c199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:57.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.544+0000 7fba40a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba3c1028b0 0x7fba3c19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:57.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.544+0000 7fba40a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba3c103270 0x7fba3c1a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:57.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.544+0000 7fba40a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c10d320 con 0x7fba3c103270 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.544+0000 7fba40a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c10d1a0 con 0x7fba3c1028b0 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.544+0000 7fba40a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c10d4a0 con 0x7fba3c1016c0 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.545+0000 7fba3be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba3c103270 0x7fba3c1a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.545+0000 7fba3be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba3c103270 0x7fba3c1a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51162/0 (socket says 172.21.15.12:51162) 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.545+0000 7fba3be00640 1 -- 172.21.15.12:0/2505458412 learned_addr learned my addr 172.21.15.12:0/2505458412 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.545+0000 7fba3aa00640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba3c1028b0 0x7fba3c19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.545+0000 7fba3b400640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c1016c0 0x7fba3c199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.545+0000 7fba3aa00640 1 -- 172.21.15.12:0/2505458412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c1016c0 msgr2=0x7fba3c199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.545+0000 7fba3aa00640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c1016c0 0x7fba3c199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.545+0000 7fba3aa00640 1 -- 172.21.15.12:0/2505458412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba3c103270 msgr2=0x7fba3c1a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.545+0000 7fba3aa00640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba3c103270 0x7fba3c1a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.545+0000 7fba3aa00640 1 -- 172.21.15.12:0/2505458412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba3c1a17a0 con 0x7fba3c1028b0 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.545+0000 7fba3be00640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba3c103270 0x7fba3c1a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.545+0000 7fba3aa00640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba3c1028b0 0x7fba3c19a0c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fba2800e9f0 tx=0x7fba2800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:57.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.546+0000 7fba33e00640 1 -- 172.21.15.12:0/2505458412 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2800ccc0 con 0x7fba3c1028b0 2024-03-17T23:54:57.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.546+0000 7fba33e00640 1 -- 172.21.15.12:0/2505458412 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba28010910 con 0x7fba3c1028b0 2024-03-17T23:54:57.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.546+0000 7fba40a00640 1 -- 172.21.15.12:0/2505458412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba3c1a1a90 con 0x7fba3c1028b0 2024-03-17T23:54:57.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.546+0000 7fba33e00640 1 -- 172.21.15.12:0/2505458412 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba28018960 con 0x7fba3c1028b0 2024-03-17T23:54:57.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.546+0000 7fba40a00640 1 -- 172.21.15.12:0/2505458412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba3c1a2030 con 0x7fba3c1028b0 2024-03-17T23:54:57.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.547+0000 7fba40a00640 1 -- 172.21.15.12:0/2505458412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba3c1a2200 con 0x7fba3c1028b0 2024-03-17T23:54:57.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.550+0000 7fba33e00640 1 -- 172.21.15.12:0/2505458412 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fba28010ac0 con 0x7fba3c1028b0 2024-03-17T23:54:57.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.551+0000 7fba33e00640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fba200a2880 0x7fba200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:57.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.552+0000 7fba33e00640 1 -- 172.21.15.12:0/2505458412 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fba280ced50 con 0x7fba3c1028b0 2024-03-17T23:54:57.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.552+0000 7fba3b400640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fba200a2880 0x7fba200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:57.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.552+0000 7fba3b400640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fba200a2880 0x7fba200a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fba3c102710 tx=0x7fba2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:57.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.556+0000 7fba33e00640 1 -- 172.21.15.12:0/2505458412 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fba2809bea0 con 0x7fba3c1028b0 2024-03-17T23:54:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:57.777+0000 7fba40a00640 1 -- 172.21.15.12:0/2505458412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fba3c1a23d0 con 0x7fba3c1028b0 2024-03-17T23:54:58.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:57 smithi191 ceph-mon[36708]: pgmap v314: 97 pgs: 7 creating+peering, 4 active+clean, 86 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:58.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:57 smithi191 ceph-mon[36708]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-17T23:54:58.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:57 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-17T23:54:58.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:57 smithi191 ceph-mon[36708]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-17T23:54:58.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:57 smithi191 ceph-mon[36708]: osdmap e85: 12 total, 12 up, 12 in 2024-03-17T23:54:58.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:57 smithi191 ceph-mon[36708]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-17T23:54:58.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:58.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:58.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:58.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:54:58.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:58.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:57 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2505458412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-17T23:54:58.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:57 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-17T23:54:58.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:57 smithi012 ceph-mon[31541]: pgmap v314: 97 pgs: 7 creating+peering, 4 active+clean, 86 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:58.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:57 smithi012 ceph-mon[31541]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-17T23:54:58.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:57 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-17T23:54:58.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:57 smithi012 ceph-mon[31541]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-17T23:54:58.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:57 smithi012 ceph-mon[31541]: osdmap e85: 12 total, 12 up, 12 in 2024-03-17T23:54:58.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:57 smithi012 ceph-mon[31541]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-17T23:54:58.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:58.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:58.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:58.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:54:58.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:58.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:57 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2505458412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-17T23:54:58.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:57 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-17T23:54:58.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:57 smithi079 ceph-mon[36241]: pgmap v314: 97 pgs: 7 creating+peering, 4 active+clean, 86 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:54:58.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:57 smithi079 ceph-mon[36241]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-17T23:54:58.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:57 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-17T23:54:58.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:57 smithi079 ceph-mon[36241]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-17T23:54:58.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:57 smithi079 ceph-mon[36241]: osdmap e85: 12 total, 12 up, 12 in 2024-03-17T23:54:58.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:57 smithi079 ceph-mon[36241]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-17T23:54:58.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:58.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:58.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:54:58.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:54:58.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:54:58.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:57 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2505458412' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-17T23:54:58.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:57 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-17T23:54:58.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.376+0000 7fba33e00640 1 -- 172.21.15.12:0/2505458412 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) ==== 157+0+0 (secure 0 0 0) 0x7fba2809b7a0 con 0x7fba3c1028b0 2024-03-17T23:54:58.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.383+0000 7fba40a00640 1 -- 172.21.15.12:0/2505458412 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fba200a2880 msgr2=0x7fba200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:58.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.383+0000 7fba40a00640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fba200a2880 0x7fba200a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fba3c102710 tx=0x7fba2c005f90 comp rx=0 tx=0).stop 2024-03-17T23:54:58.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.383+0000 7fba40a00640 1 -- 172.21.15.12:0/2505458412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba3c1028b0 msgr2=0x7fba3c19a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:58.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.383+0000 7fba40a00640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba3c1028b0 0x7fba3c19a0c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fba2800e9f0 tx=0x7fba2800eeb0 comp rx=0 tx=0).stop 2024-03-17T23:54:58.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.384+0000 7fba40a00640 1 -- 172.21.15.12:0/2505458412 shutdown_connections 2024-03-17T23:54:58.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.384+0000 7fba40a00640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fba200a2880 0x7fba200a4d40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:58.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.384+0000 7fba40a00640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba3c103270 0x7fba3c1a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:58.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.384+0000 7fba40a00640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba3c1028b0 0x7fba3c19a0c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:58.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.384+0000 7fba40a00640 1 --2- 172.21.15.12:0/2505458412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c1016c0 0x7fba3c199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:58.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.384+0000 7fba40a00640 1 -- 172.21.15.12:0/2505458412 >> 172.21.15.12:0/2505458412 conn(0x7fba3c0fcc50 msgr2=0x7fba3c073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:58.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.384+0000 7fba40a00640 1 -- 172.21.15.12:0/2505458412 shutdown_connections 2024-03-17T23:54:58.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.384+0000 7fba40a00640 1 -- 172.21.15.12:0/2505458412 wait complete. 2024-03-17T23:54:58.409 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-17T23:54:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.623+0000 7f665ca00640 1 -- 172.21.15.12:0/109404300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6658108f70 msgr2=0x7f66581093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.623+0000 7f665ca00640 1 --2- 172.21.15.12:0/109404300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6658108f70 0x7f66581093f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6640009900 tx=0x7f664002bc30 comp rx=0 tx=0).stop 2024-03-17T23:54:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.623+0000 7f665ca00640 1 -- 172.21.15.12:0/109404300 shutdown_connections 2024-03-17T23:54:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.623+0000 7f665ca00640 1 --2- 172.21.15.12:0/109404300 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6658109930 0x7f6658110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.623+0000 7f665ca00640 1 --2- 172.21.15.12:0/109404300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6658108f70 0x7f66581093f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.623+0000 7f665ca00640 1 --2- 172.21.15.12:0/109404300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6658107e20 0x7f6658108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.623+0000 7f665ca00640 1 -- 172.21.15.12:0/109404300 >> 172.21.15.12:0/109404300 conn(0x7f6658076df0 msgr2=0x7f6658079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:54:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.623+0000 7f665ca00640 1 -- 172.21.15.12:0/109404300 shutdown_connections 2024-03-17T23:54:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.623+0000 7f665ca00640 1 -- 172.21.15.12:0/109404300 wait complete. 2024-03-17T23:54:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.625+0000 7f665ca00640 1 Processor -- start 2024-03-17T23:54:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.625+0000 7f665ca00640 1 -- start start 2024-03-17T23:54:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.625+0000 7f665ca00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6658107e20 0x7f665819e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.625+0000 7f665ca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6658108f70 0x7f665819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.625+0000 7f665ca00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6658109930 0x7f66581a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.625+0000 7f665ca00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6658113af0 con 0x7f6658108f70 2024-03-17T23:54:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.625+0000 7f665ca00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6658113970 con 0x7f6658109930 2024-03-17T23:54:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.625+0000 7f665ca00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6658113c70 con 0x7f6658107e20 2024-03-17T23:54:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.626+0000 7f6656800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6658109930 0x7f66581a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.626+0000 7f6656800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6658109930 0x7f66581a56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:49612/0 (socket says 172.21.15.12:49612) 2024-03-17T23:54:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.626+0000 7f6656800640 1 -- 172.21.15.12:0/1803481318 learned_addr learned my addr 172.21.15.12:0/1803481318 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:54:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.626+0000 7f664fe00640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6658108f70 0x7f665819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.626+0000 7f6655e00640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6658107e20 0x7f665819e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.626+0000 7f6656800640 1 -- 172.21.15.12:0/1803481318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6658107e20 msgr2=0x7f665819e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.626+0000 7f6656800640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6658107e20 0x7f665819e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.626+0000 7f6656800640 1 -- 172.21.15.12:0/1803481318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6658108f70 msgr2=0x7f665819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:54:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.626+0000 7f6656800640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6658108f70 0x7f665819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:54:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.626+0000 7f6656800640 1 -- 172.21.15.12:0/1803481318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66581a5d20 con 0x7f6658109930 2024-03-17T23:54:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.626+0000 7f664fe00640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6658108f70 0x7f665819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:54:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.626+0000 7f6656800640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6658109930 0x7f66581a56b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f665000eb10 tx=0x7f665000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.626+0000 7f664ea00640 1 -- 172.21.15.12:0/1803481318 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6650004b20 con 0x7f6658109930 2024-03-17T23:54:58.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.627+0000 7f664ea00640 1 -- 172.21.15.12:0/1803481318 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6650004cc0 con 0x7f6658109930 2024-03-17T23:54:58.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.627+0000 7f664ea00640 1 -- 172.21.15.12:0/1803481318 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6650010760 con 0x7f6658109930 2024-03-17T23:54:58.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.627+0000 7f665ca00640 1 -- 172.21.15.12:0/1803481318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66581a5fb0 con 0x7f6658109930 2024-03-17T23:54:58.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.627+0000 7f665ca00640 1 -- 172.21.15.12:0/1803481318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66581a6480 con 0x7f6658109930 2024-03-17T23:54:58.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.628+0000 7f665ca00640 1 -- 172.21.15.12:0/1803481318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f665810f0d0 con 0x7f6658109930 2024-03-17T23:54:58.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.630+0000 7f664ea00640 1 -- 172.21.15.12:0/1803481318 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6650005020 con 0x7f6658109930 2024-03-17T23:54:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.631+0000 7f664ea00640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f663c0a2800 0x7f663c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:54:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.632+0000 7f664ea00640 1 -- 172.21.15.12:0/1803481318 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8277+0+0 (secure 0 0 0) 0x7f66500cf0b0 con 0x7f6658109930 2024-03-17T23:54:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.632+0000 7f6655e00640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f663c0a2800 0x7f663c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:54:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.632+0000 7f6655e00640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f663c0a2800 0x7f663c0a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6648009bd0 tx=0x7f6648005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:54:58.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.635+0000 7f664ea00640 1 -- 172.21.15.12:0/1803481318 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f665009c120 con 0x7f6658109930 2024-03-17T23:54:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:54:58.846+0000 7f665ca00640 1 -- 172.21.15.12:0/1803481318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f66580623e0 con 0x7f6658109930 2024-03-17T23:54:59.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:59 smithi191 ceph-mon[36708]: pgmap v317: 97 pgs: 7 creating+peering, 33 active+clean, 57 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-17T23:54:59.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:59 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] up:active 2024-03-17T23:54:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:59 smithi191 ceph-mon[36708]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:54:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:59 smithi191 ceph-mon[36708]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:54:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:59 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-17T23:54:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:59 smithi191 ceph-mon[36708]: osdmap e86: 12 total, 12 up, 12 in 2024-03-17T23:54:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:59 smithi191 ceph-mon[36708]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-17T23:54:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:59 smithi191 ceph-mon[36708]: Cluster is now healthy 2024-03-17T23:54:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:59 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1803481318' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-17T23:54:59.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:54:59 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-17T23:54:59.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:59 smithi012 ceph-mon[31541]: pgmap v317: 97 pgs: 7 creating+peering, 33 active+clean, 57 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-17T23:54:59.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:59 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] up:active 2024-03-17T23:54:59.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:59 smithi012 ceph-mon[31541]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:54:59.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:59 smithi012 ceph-mon[31541]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:54:59.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:59 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-17T23:54:59.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:59 smithi012 ceph-mon[31541]: osdmap e86: 12 total, 12 up, 12 in 2024-03-17T23:54:59.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:59 smithi012 ceph-mon[31541]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-17T23:54:59.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:59 smithi012 ceph-mon[31541]: Cluster is now healthy 2024-03-17T23:54:59.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:59 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1803481318' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-17T23:54:59.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:54:59 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-17T23:54:59.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:59 smithi079 ceph-mon[36241]: pgmap v317: 97 pgs: 7 creating+peering, 33 active+clean, 57 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-17T23:54:59.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:59 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] up:active 2024-03-17T23:54:59.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:59 smithi079 ceph-mon[36241]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:54:59.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:59 smithi079 ceph-mon[36241]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:54:59.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:59 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-17T23:54:59.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:59 smithi079 ceph-mon[36241]: osdmap e86: 12 total, 12 up, 12 in 2024-03-17T23:54:59.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:59 smithi079 ceph-mon[36241]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-17T23:54:59.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:59 smithi079 ceph-mon[36241]: Cluster is now healthy 2024-03-17T23:54:59.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:59 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1803481318' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-17T23:54:59.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:54:59 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-17T23:55:00.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:00.838+0000 7f664ea00640 1 -- 172.21.15.12:0/1803481318 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) ==== 228+0+0 (secure 0 0 0) 0x7f665009ba20 con 0x7f6658109930 2024-03-17T23:55:00.840 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data_ec' created 2024-03-17T23:55:00.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:00.844+0000 7f665ca00640 1 -- 172.21.15.12:0/1803481318 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f663c0a2800 msgr2=0x7f663c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:00.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:00.844+0000 7f665ca00640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f663c0a2800 0x7f663c0a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6648009bd0 tx=0x7f6648005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:00.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:00.844+0000 7f665ca00640 1 -- 172.21.15.12:0/1803481318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6658109930 msgr2=0x7f66581a56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:00.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:00.844+0000 7f665ca00640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6658109930 0x7f66581a56b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f665000eb10 tx=0x7f665000c2d0 comp rx=0 tx=0).stop 2024-03-17T23:55:00.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:00.844+0000 7f665ca00640 1 -- 172.21.15.12:0/1803481318 shutdown_connections 2024-03-17T23:55:00.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:00.844+0000 7f665ca00640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f663c0a2800 0x7f663c0a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:00.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:00.844+0000 7f665ca00640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6658109930 0x7f66581a56b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:00.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:00.844+0000 7f665ca00640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6658108f70 0x7f665819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:00.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:00.844+0000 7f665ca00640 1 --2- 172.21.15.12:0/1803481318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6658107e20 0x7f665819e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:00.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:00.844+0000 7f665ca00640 1 -- 172.21.15.12:0/1803481318 >> 172.21.15.12:0/1803481318 conn(0x7f6658076df0 msgr2=0x7f665810c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:00.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:00.847+0000 7f665ca00640 1 -- 172.21.15.12:0/1803481318 shutdown_connections 2024-03-17T23:55:00.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:00.848+0000 7f665ca00640 1 -- 172.21.15.12:0/1803481318 wait complete. 2024-03-17T23:55:00.875 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-17T23:55:01.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.072+0000 7f3b82400640 1 -- 172.21.15.12:0/1247418896 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b7c1049f0 msgr2=0x7f3b7c104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:01.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.072+0000 7f3b82400640 1 --2- 172.21.15.12:0/1247418896 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b7c1049f0 0x7f3b7c104e70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c009900 tx=0x7f3b6c02bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:01.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.072+0000 7f3b82400640 1 -- 172.21.15.12:0/1247418896 shutdown_connections 2024-03-17T23:55:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.072+0000 7f3b82400640 1 --2- 172.21.15.12:0/1247418896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b7c1053b0 0x7f3b7c10bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.072+0000 7f3b82400640 1 --2- 172.21.15.12:0/1247418896 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b7c1049f0 0x7f3b7c104e70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.072+0000 7f3b82400640 1 --2- 172.21.15.12:0/1247418896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b7c103800 0x7f3b7c103c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.072+0000 7f3b82400640 1 -- 172.21.15.12:0/1247418896 >> 172.21.15.12:0/1247418896 conn(0x7f3b7c0fed70 msgr2=0x7f3b7c1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.072+0000 7f3b82400640 1 -- 172.21.15.12:0/1247418896 shutdown_connections 2024-03-17T23:55:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.072+0000 7f3b82400640 1 -- 172.21.15.12:0/1247418896 wait complete. 2024-03-17T23:55:01.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.074+0000 7f3b82400640 1 Processor -- start 2024-03-17T23:55:01.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.074+0000 7f3b82400640 1 -- start start 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.074+0000 7f3b82400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b7c103800 0x7f3b7c199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.074+0000 7f3b82400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b7c1049f0 0x7f3b7c19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.074+0000 7f3b82400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b7c1053b0 0x7f3b7c1a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.074+0000 7f3b82400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c10f480 con 0x7f3b7c1053b0 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.074+0000 7f3b82400640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c10f300 con 0x7f3b7c103800 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.074+0000 7f3b82400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c10f600 con 0x7f3b7c1049f0 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.075+0000 7f3b81000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b7c103800 0x7f3b7c199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.075+0000 7f3b7be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b7c1049f0 0x7f3b7c19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.075+0000 7f3b81000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b7c103800 0x7f3b7c199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:49636/0 (socket says 172.21.15.12:49636) 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.075+0000 7f3b81000640 1 -- 172.21.15.12:0/2198603950 learned_addr learned my addr 172.21.15.12:0/2198603950 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.075+0000 7f3b81a00640 1 --2- 172.21.15.12:0/2198603950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b7c1053b0 0x7f3b7c1a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.075+0000 7f3b81000640 1 -- 172.21.15.12:0/2198603950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b7c1049f0 msgr2=0x7f3b7c19a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.075+0000 7f3b81000640 1 --2- 172.21.15.12:0/2198603950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b7c1049f0 0x7f3b7c19a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.075+0000 7f3b81000640 1 -- 172.21.15.12:0/2198603950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b7c1053b0 msgr2=0x7f3b7c1a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.075+0000 7f3b81000640 1 --2- 172.21.15.12:0/2198603950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b7c1053b0 0x7f3b7c1a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.075+0000 7f3b81000640 1 -- 172.21.15.12:0/2198603950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7c1a16f0 con 0x7f3b7c103800 2024-03-17T23:55:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.075+0000 7f3b81000640 1 --2- 172.21.15.12:0/2198603950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b7c103800 0x7f3b7c199ad0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000e9c0 tx=0x7f3b7000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.075+0000 7f3b79600640 1 -- 172.21.15.12:0/2198603950 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7000cd80 con 0x7f3b7c103800 2024-03-17T23:55:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.075+0000 7f3b79600640 1 -- 172.21.15.12:0/2198603950 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b700035f0 con 0x7f3b7c103800 2024-03-17T23:55:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.076+0000 7f3b82400640 1 -- 172.21.15.12:0/2198603950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b7c1a19e0 con 0x7f3b7c103800 2024-03-17T23:55:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.076+0000 7f3b79600640 1 -- 172.21.15.12:0/2198603950 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b70010690 con 0x7f3b7c103800 2024-03-17T23:55:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.076+0000 7f3b82400640 1 -- 172.21.15.12:0/2198603950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b7c1a1f00 con 0x7f3b7c103800 2024-03-17T23:55:01.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.077+0000 7f3b82400640 1 -- 172.21.15.12:0/2198603950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b7c068940 con 0x7f3b7c103800 2024-03-17T23:55:01.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.078+0000 7f3b79600640 1 -- 172.21.15.12:0/2198603950 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3b700037a0 con 0x7f3b7c103800 2024-03-17T23:55:01.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.080+0000 7f3b79600640 1 --2- 172.21.15.12:0/2198603950 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3b640a2880 0x7f3b640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:01.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.080+0000 7f3b79600640 1 -- 172.21.15.12:0/2198603950 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f3b700cefb0 con 0x7f3b7c103800 2024-03-17T23:55:01.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.080+0000 7f3b7be00640 1 --2- 172.21.15.12:0/2198603950 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3b640a2880 0x7f3b640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:01.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.081+0000 7f3b7be00640 1 --2- 172.21.15.12:0/2198603950 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3b640a2880 0x7f3b640a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c02c7b0 tx=0x7f3b6c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:01.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.084+0000 7f3b79600640 1 -- 172.21.15.12:0/2198603950 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3b7009bdd0 con 0x7f3b7c103800 2024-03-17T23:55:01.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:00 smithi191 ceph-mon[36708]: osdmap e87: 12 total, 12 up, 12 in 2024-03-17T23:55:01.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:00 smithi012 ceph-mon[31541]: osdmap e87: 12 total, 12 up, 12 in 2024-03-17T23:55:01.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.283+0000 7f3b82400640 1 -- 172.21.15.12:0/2198603950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f3b7c10ab20 con 0x7f3b7c103800 2024-03-17T23:55:01.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:00 smithi079 ceph-mon[36241]: osdmap e87: 12 total, 12 up, 12 in 2024-03-17T23:55:01.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.855+0000 7f3b79600640 1 -- 172.21.15.12:0/2198603950 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) ==== 169+0+0 (secure 0 0 0) 0x7f3b7009b6d0 con 0x7f3b7c103800 2024-03-17T23:55:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:set pool 4 allow_ec_overwrites to true 2024-03-17T23:55:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.863+0000 7f3b82400640 1 -- 172.21.15.12:0/2198603950 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3b640a2880 msgr2=0x7f3b640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.863+0000 7f3b82400640 1 --2- 172.21.15.12:0/2198603950 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3b640a2880 0x7f3b640a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c02c7b0 tx=0x7f3b6c005fd0 comp rx=0 tx=0).stop 2024-03-17T23:55:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.863+0000 7f3b82400640 1 -- 172.21.15.12:0/2198603950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b7c103800 msgr2=0x7f3b7c199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.863+0000 7f3b82400640 1 --2- 172.21.15.12:0/2198603950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b7c103800 0x7f3b7c199ad0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000e9c0 tx=0x7f3b7000ee80 comp rx=0 tx=0).stop 2024-03-17T23:55:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.864+0000 7f3b82400640 1 -- 172.21.15.12:0/2198603950 shutdown_connections 2024-03-17T23:55:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.864+0000 7f3b82400640 1 --2- 172.21.15.12:0/2198603950 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3b640a2880 0x7f3b640a4d40 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c02c7b0 tx=0x7f3b6c005fd0 comp rx=0 tx=0).stop 2024-03-17T23:55:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.864+0000 7f3b82400640 1 --2- 172.21.15.12:0/2198603950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b7c1053b0 0x7f3b7c1a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.864+0000 7f3b82400640 1 --2- 172.21.15.12:0/2198603950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b7c1049f0 0x7f3b7c19a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.864+0000 7f3b82400640 1 --2- 172.21.15.12:0/2198603950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b7c103800 0x7f3b7c199ad0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.864+0000 7f3b82400640 1 -- 172.21.15.12:0/2198603950 >> 172.21.15.12:0/2198603950 conn(0x7f3b7c0fed70 msgr2=0x7f3b7c107cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.865+0000 7f3b82400640 1 -- 172.21.15.12:0/2198603950 shutdown_connections 2024-03-17T23:55:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:01.865+0000 7f3b82400640 1 -- 172.21.15.12:0/2198603950 wait complete. 2024-03-17T23:55:01.899 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-17T23:55:01.937 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 17 23:55:01 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b[66285]: 2024-03-17T23:55:01.674+0000 7ff92867c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-17T23:55:01.937 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:01 smithi079 ceph-mon[36241]: pgmap v320: 97 pgs: 4 creating+peering, 93 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-17T23:55:01.938 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:01 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-17T23:55:01.938 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:01 smithi079 ceph-mon[36241]: osdmap e88: 12 total, 12 up, 12 in 2024-03-17T23:55:01.938 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:01 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2198603950' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-17T23:55:01.938 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:01 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-17T23:55:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.082+0000 7f86dea00640 1 -- 172.21.15.12:0/4113759820 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86d81022b0 msgr2=0x7f86d8108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.082+0000 7f86dea00640 1 --2- 172.21.15.12:0/4113759820 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86d81022b0 0x7f86d8108b40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f86d0009ca0 tx=0x7f86d002c0f0 comp rx=0 tx=0).stop 2024-03-17T23:55:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.082+0000 7f86dea00640 1 -- 172.21.15.12:0/4113759820 shutdown_connections 2024-03-17T23:55:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.082+0000 7f86dea00640 1 --2- 172.21.15.12:0/4113759820 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86d81022b0 0x7f86d8108b40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.082+0000 7f86dea00640 1 --2- 172.21.15.12:0/4113759820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86d81018f0 0x7f86d8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.082+0000 7f86dea00640 1 --2- 172.21.15.12:0/4113759820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d8100700 0x7f86d8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.082+0000 7f86dea00640 1 -- 172.21.15.12:0/4113759820 >> 172.21.15.12:0/4113759820 conn(0x7f86d80fbcb0 msgr2=0x7f86d80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.083+0000 7f86dea00640 1 -- 172.21.15.12:0/4113759820 shutdown_connections 2024-03-17T23:55:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.083+0000 7f86dea00640 1 -- 172.21.15.12:0/4113759820 wait complete. 2024-03-17T23:55:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.084+0000 7f86dea00640 1 Processor -- start 2024-03-17T23:55:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.084+0000 7f86dea00640 1 -- start start 2024-03-17T23:55:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.084+0000 7f86dea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86d8100700 0x7f86d810b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.084+0000 7f86dea00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86d81018f0 0x7f86d81096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.084+0000 7f86dea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d81022b0 0x7f86d8109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.084+0000 7f86dea00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f86d810d4a0 con 0x7f86d81022b0 2024-03-17T23:55:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.084+0000 7f86dea00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f86d810d320 con 0x7f86d81018f0 2024-03-17T23:55:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.084+0000 7f86dea00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f86d810d620 con 0x7f86d8100700 2024-03-17T23:55:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.084+0000 7f86d7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86d8100700 0x7f86d810b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.085+0000 7f86dca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d81022b0 0x7f86d8109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.085+0000 7f86d7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86d8100700 0x7f86d810b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:51766/0 (socket says 172.21.15.12:51766) 2024-03-17T23:55:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.085+0000 7f86d7e00640 1 -- 172.21.15.12:0/2319885377 learned_addr learned my addr 172.21.15.12:0/2319885377 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.085+0000 7f86cfe00640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86d81018f0 0x7f86d81096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.085+0000 7f86dca00640 1 -- 172.21.15.12:0/2319885377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86d8100700 msgr2=0x7f86d810b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.085+0000 7f86dca00640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86d8100700 0x7f86d810b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.085+0000 7f86dca00640 1 -- 172.21.15.12:0/2319885377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86d81018f0 msgr2=0x7f86d81096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.085+0000 7f86dca00640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86d81018f0 0x7f86d81096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.085+0000 7f86dca00640 1 -- 172.21.15.12:0/2319885377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d810a490 con 0x7f86d81022b0 2024-03-17T23:55:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.085+0000 7f86cfe00640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86d81018f0 0x7f86d81096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:55:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.085+0000 7f86dca00640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d81022b0 0x7f86d8109c00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f86d0009bf0 tx=0x7f86d0032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.086+0000 7f86d5600640 1 -- 172.21.15.12:0/2319885377 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d0041020 con 0x7f86d81022b0 2024-03-17T23:55:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.086+0000 7f86d5600640 1 -- 172.21.15.12:0/2319885377 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86d0034cd0 con 0x7f86d81022b0 2024-03-17T23:55:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.086+0000 7f86dea00640 1 -- 172.21.15.12:0/2319885377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86d81a1b60 con 0x7f86d81022b0 2024-03-17T23:55:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.086+0000 7f86d5600640 1 -- 172.21.15.12:0/2319885377 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d0039040 con 0x7f86d81022b0 2024-03-17T23:55:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.086+0000 7f86dea00640 1 -- 172.21.15.12:0/2319885377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86d81a2080 con 0x7f86d81022b0 2024-03-17T23:55:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.087+0000 7f86dea00640 1 -- 172.21.15.12:0/2319885377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86d8068a20 con 0x7f86d81022b0 2024-03-17T23:55:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.089+0000 7f86d5600640 1 -- 172.21.15.12:0/2319885377 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f86d0044020 con 0x7f86d81022b0 2024-03-17T23:55:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.090+0000 7f86d5600640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f86bc0a2800 0x7f86bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.091+0000 7f86d5600640 1 -- 172.21.15.12:0/2319885377 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7f86d00f0f30 con 0x7f86d81022b0 2024-03-17T23:55:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.091+0000 7f86d7e00640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f86bc0a2800 0x7f86bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:02.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:01 smithi191 ceph-mon[36708]: pgmap v320: 97 pgs: 4 creating+peering, 93 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-17T23:55:02.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:01 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-17T23:55:02.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:01 smithi191 ceph-mon[36708]: osdmap e88: 12 total, 12 up, 12 in 2024-03-17T23:55:02.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:01 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2198603950' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-17T23:55:02.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:01 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-17T23:55:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.091+0000 7f86d7e00640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f86bc0a2800 0x7f86bc0a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f86c80058c0 tx=0x7f86c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.093+0000 7f86d5600640 1 -- 172.21.15.12:0/2319885377 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f86d004b020 con 0x7f86d81022b0 2024-03-17T23:55:02.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:01 smithi012 ceph-mon[31541]: pgmap v320: 97 pgs: 4 creating+peering, 93 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-17T23:55:02.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:01 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-17T23:55:02.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:01 smithi012 ceph-mon[31541]: osdmap e88: 12 total, 12 up, 12 in 2024-03-17T23:55:02.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:01 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2198603950' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-17T23:55:02.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:01 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-17T23:55:02.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.335+0000 7f86dea00640 1 -- 172.21.15.12:0/2319885377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f86d8107a10 con 0x7f86d81022b0 2024-03-17T23:55:02.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.864+0000 7f86d5600640 1 -- 172.21.15.12:0/2319885377 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f86d8107a10 con 0x7f86d81022b0 2024-03-17T23:55:02.866 INFO:teuthology.orchestra.run.smithi012.stderr:added data pool 4 to fsmap 2024-03-17T23:55:02.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.869+0000 7f86dea00640 1 -- 172.21.15.12:0/2319885377 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f86bc0a2800 msgr2=0x7f86bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:02.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.869+0000 7f86dea00640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f86bc0a2800 0x7f86bc0a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f86c80058c0 tx=0x7f86c8005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:02.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.869+0000 7f86dea00640 1 -- 172.21.15.12:0/2319885377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d81022b0 msgr2=0x7f86d8109c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:02.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.869+0000 7f86dea00640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d81022b0 0x7f86d8109c00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f86d0009bf0 tx=0x7f86d0032ea0 comp rx=0 tx=0).stop 2024-03-17T23:55:02.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.869+0000 7f86dea00640 1 -- 172.21.15.12:0/2319885377 shutdown_connections 2024-03-17T23:55:02.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.869+0000 7f86dea00640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f86bc0a2800 0x7f86bc0a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:02.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.869+0000 7f86dea00640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86d81022b0 0x7f86d8109c00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:02.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.869+0000 7f86dea00640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86d81018f0 0x7f86d81096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:02.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.869+0000 7f86dea00640 1 --2- 172.21.15.12:0/2319885377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86d8100700 0x7f86d810b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:02.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.869+0000 7f86dea00640 1 -- 172.21.15.12:0/2319885377 >> 172.21.15.12:0/2319885377 conn(0x7f86d80fbcb0 msgr2=0x7f86d8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:02.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.869+0000 7f86dea00640 1 -- 172.21.15.12:0/2319885377 shutdown_connections 2024-03-17T23:55:02.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:02.869+0000 7f86dea00640 1 -- 172.21.15.12:0/2319885377 wait complete. 2024-03-17T23:55:02.890 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:55:03.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.108+0000 7f19d7e00640 1 -- 172.21.15.12:0/32693824 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19d81027d0 msgr2=0x7f19d8102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:03.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.108+0000 7f19d7e00640 1 --2- 172.21.15.12:0/32693824 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19d81027d0 0x7f19d8102c50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f19c0009900 tx=0x7f19c002bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:03.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.109+0000 7f19d7e00640 1 -- 172.21.15.12:0/32693824 shutdown_connections 2024-03-17T23:55:03.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.109+0000 7f19d7e00640 1 --2- 172.21.15.12:0/32693824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19d8103190 0x7f19d8109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.109+0000 7f19d7e00640 1 --2- 172.21.15.12:0/32693824 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19d81027d0 0x7f19d8102c50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.109+0000 7f19d7e00640 1 --2- 172.21.15.12:0/32693824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19d81015e0 0x7f19d8101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.109+0000 7f19d7e00640 1 -- 172.21.15.12:0/32693824 >> 172.21.15.12:0/32693824 conn(0x7f19d80fcb70 msgr2=0x7f19d80fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:03.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.109+0000 7f19d7e00640 1 -- 172.21.15.12:0/32693824 shutdown_connections 2024-03-17T23:55:03.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.109+0000 7f19d7e00640 1 -- 172.21.15.12:0/32693824 wait complete. 2024-03-17T23:55:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.110+0000 7f19d7e00640 1 Processor -- start 2024-03-17T23:55:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.110+0000 7f19d7e00640 1 -- start start 2024-03-17T23:55:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.110+0000 7f19d7e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19d81015e0 0x7f19d8199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.110+0000 7f19d7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19d81027d0 0x7f19d819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.110+0000 7f19d7e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19d8103190 0x7f19d81a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.110+0000 7f19d7e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f19d810d2a0 con 0x7f19d81015e0 2024-03-17T23:55:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.110+0000 7f19d7e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f19d810d120 con 0x7f19d8103190 2024-03-17T23:55:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.110+0000 7f19d7e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f19d810d420 con 0x7f19d81027d0 2024-03-17T23:55:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.110+0000 7f19d7400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19d8103190 0x7f19d81a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.110+0000 7f19d6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19d81015e0 0x7f19d8199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.110+0000 7f19d7400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19d8103190 0x7f19d81a1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:49672/0 (socket says 172.21.15.12:49672) 2024-03-17T23:55:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.110+0000 7f19d7400640 1 -- 172.21.15.12:0/3997013588 learned_addr learned my addr 172.21.15.12:0/3997013588 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.111+0000 7f19cfe00640 1 --2- 172.21.15.12:0/3997013588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19d81027d0 0x7f19d819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.111+0000 7f19d7400640 1 -- 172.21.15.12:0/3997013588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19d81027d0 msgr2=0x7f19d819a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.111+0000 7f19d7400640 1 --2- 172.21.15.12:0/3997013588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19d81027d0 0x7f19d819a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.111+0000 7f19d7400640 1 -- 172.21.15.12:0/3997013588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19d81015e0 msgr2=0x7f19d8199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.111+0000 7f19d7400640 1 --2- 172.21.15.12:0/3997013588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19d81015e0 0x7f19d8199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.111+0000 7f19d7400640 1 -- 172.21.15.12:0/3997013588 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d81a16f0 con 0x7f19d8103190 2024-03-17T23:55:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.111+0000 7f19d7400640 1 --2- 172.21.15.12:0/3997013588 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19d8103190 0x7f19d81a1080 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f19d000eaa0 tx=0x7f19d000ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.111+0000 7f19dca00640 1 -- 172.21.15.12:0/3997013588 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d000cdb0 con 0x7f19d8103190 2024-03-17T23:55:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.112+0000 7f19dca00640 1 -- 172.21.15.12:0/3997013588 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19d0005020 con 0x7f19d8103190 2024-03-17T23:55:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.112+0000 7f19dca00640 1 -- 172.21.15.12:0/3997013588 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d00116b0 con 0x7f19d8103190 2024-03-17T23:55:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.112+0000 7f19d7e00640 1 -- 172.21.15.12:0/3997013588 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d81a19e0 con 0x7f19d8103190 2024-03-17T23:55:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.112+0000 7f19d7e00640 1 -- 172.21.15.12:0/3997013588 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d81a1eb0 con 0x7f19d8103190 2024-03-17T23:55:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.113+0000 7f19d7e00640 1 -- 172.21.15.12:0/3997013588 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d81a20e0 con 0x7f19d8103190 2024-03-17T23:55:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.114+0000 7f19dca00640 1 -- 172.21.15.12:0/3997013588 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f19d0005560 con 0x7f19d8103190 2024-03-17T23:55:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.116+0000 7f19dca00640 1 --2- 172.21.15.12:0/3997013588 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19bc0a2800 0x7f19bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.116+0000 7f19dca00640 1 -- 172.21.15.12:0/3997013588 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f19d00ceeb0 con 0x7f19d8103190 2024-03-17T23:55:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.116+0000 7f19d6a00640 1 --2- 172.21.15.12:0/3997013588 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19bc0a2800 0x7f19bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.117+0000 7f19d6a00640 1 --2- 172.21.15.12:0/3997013588 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19bc0a2800 0x7f19bc0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f19d8102630 tx=0x7f19c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.120+0000 7f19dca00640 1 -- 172.21.15.12:0/3997013588 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f19d0016030 con 0x7f19d8103190 2024-03-17T23:55:03.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:02 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-17T23:55:03.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:02 smithi012 ceph-mon[31541]: osdmap e89: 12 total, 12 up, 12 in 2024-03-17T23:55:03.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:02 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2319885377' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-17T23:55:03.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:02 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-17T23:55:03.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:02 smithi079 ceph-mon[36241]: osdmap e89: 12 total, 12 up, 12 in 2024-03-17T23:55:03.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:02 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2319885377' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-17T23:55:03.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:02 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-17T23:55:03.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:02 smithi191 ceph-mon[36708]: osdmap e89: 12 total, 12 up, 12 in 2024-03-17T23:55:03.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:02 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2319885377' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-17T23:55:03.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.364+0000 7f19d7e00640 1 -- 172.21.15.12:0/3997013588 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19d81a22b0 con 0x7f19d8103190 2024-03-17T23:55:03.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.365+0000 7f19dca00640 1 -- 172.21.15.12:0/3997013588 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11800 (secure 0 0 0) 0x7f19d009b660 con 0x7f19d8103190 2024-03-17T23:55:03.366 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:03.366 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/2073607248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":2073607248},{"type":"v1","addr":"172.21.15.12:6841","nonce":2073607248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:54:56.312554+0000","modified":"2024-03-17T23:55:02.337827+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/2478742711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":2478742711},{"type":"v1","addr":"172.21.15.79:6839","nonce":2478742711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-03-17T23:55:03.367 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 17 2024-03-17T23:55:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.368+0000 7f19d7e00640 1 -- 172.21.15.12:0/3997013588 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19bc0a2800 msgr2=0x7f19bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.368+0000 7f19d7e00640 1 --2- 172.21.15.12:0/3997013588 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19bc0a2800 0x7f19bc0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f19d8102630 tx=0x7f19c8005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.368+0000 7f19d7e00640 1 -- 172.21.15.12:0/3997013588 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19d8103190 msgr2=0x7f19d81a1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.368+0000 7f19d7e00640 1 --2- 172.21.15.12:0/3997013588 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19d8103190 0x7f19d81a1080 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f19d000eaa0 tx=0x7f19d000ef60 comp rx=0 tx=0).stop 2024-03-17T23:55:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.368+0000 7f19d7e00640 1 -- 172.21.15.12:0/3997013588 shutdown_connections 2024-03-17T23:55:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.368+0000 7f19d7e00640 1 --2- 172.21.15.12:0/3997013588 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19bc0a2800 0x7f19bc0a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.368+0000 7f19d7e00640 1 --2- 172.21.15.12:0/3997013588 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19d8103190 0x7f19d81a1080 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.368+0000 7f19d7e00640 1 --2- 172.21.15.12:0/3997013588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19d81027d0 0x7f19d819a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.368+0000 7f19d7e00640 1 --2- 172.21.15.12:0/3997013588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19d81015e0 0x7f19d8199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.368+0000 7f19d7e00640 1 -- 172.21.15.12:0/3997013588 >> 172.21.15.12:0/3997013588 conn(0x7f19d80fcb70 msgr2=0x7f19d8073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:03.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.368+0000 7f19d7e00640 1 -- 172.21.15.12:0/3997013588 shutdown_connections 2024-03-17T23:55:03.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.369+0000 7f19d7e00640 1 -- 172.21.15.12:0/3997013588 wait complete. 2024-03-17T23:55:03.392 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:55:03.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.598+0000 7f03e5400640 1 -- 172.21.15.12:0/2149551120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03e01055b0 msgr2=0x7f03e010be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:03.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.598+0000 7f03e5400640 1 --2- 172.21.15.12:0/2149551120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03e01055b0 0x7f03e010be40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009900 tx=0x7f03d802bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:03.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.598+0000 7f03e5400640 1 -- 172.21.15.12:0/2149551120 shutdown_connections 2024-03-17T23:55:03.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.598+0000 7f03e5400640 1 --2- 172.21.15.12:0/2149551120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03e01055b0 0x7f03e010be40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.598+0000 7f03e5400640 1 --2- 172.21.15.12:0/2149551120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03e0104bf0 0x7f03e0105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.598+0000 7f03e5400640 1 --2- 172.21.15.12:0/2149551120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e0103a00 0x7f03e0103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.598+0000 7f03e5400640 1 -- 172.21.15.12:0/2149551120 >> 172.21.15.12:0/2149551120 conn(0x7f03e00fef90 msgr2=0x7f03e01013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:03.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.598+0000 7f03e5400640 1 -- 172.21.15.12:0/2149551120 shutdown_connections 2024-03-17T23:55:03.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.598+0000 7f03e5400640 1 -- 172.21.15.12:0/2149551120 wait complete. 2024-03-17T23:55:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.599+0000 7f03e5400640 1 Processor -- start 2024-03-17T23:55:03.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.599+0000 7f03e5400640 1 -- start start 2024-03-17T23:55:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03e5400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e0103a00 0x7f03e0199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03e5400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03e0104bf0 0x7f03e019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03e5400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03e01055b0 0x7f03e01a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03e5400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f03e010f6d0 con 0x7f03e0103a00 2024-03-17T23:55:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03e5400640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f03e010f550 con 0x7f03e0104bf0 2024-03-17T23:55:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03e5400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f03e010f850 con 0x7f03e01055b0 2024-03-17T23:55:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03d5e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03e0104bf0 0x7f03e019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03de800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e0103a00 0x7f03e0199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03d5e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03e0104bf0 0x7f03e019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:49696/0 (socket says 172.21.15.12:49696) 2024-03-17T23:55:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03d5e00640 1 -- 172.21.15.12:0/51153116 learned_addr learned my addr 172.21.15.12:0/51153116 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03df200640 1 --2- 172.21.15.12:0/51153116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03e01055b0 0x7f03e01a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:03.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03d5e00640 1 -- 172.21.15.12:0/51153116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03e01055b0 msgr2=0x7f03e01a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03d5e00640 1 --2- 172.21.15.12:0/51153116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03e01055b0 0x7f03e01a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03d5e00640 1 -- 172.21.15.12:0/51153116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e0103a00 msgr2=0x7f03e0199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03d5e00640 1 --2- 172.21.15.12:0/51153116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e0103a00 0x7f03e0199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03d5e00640 1 -- 172.21.15.12:0/51153116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e01a18d0 con 0x7f03e0104bf0 2024-03-17T23:55:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.600+0000 7f03d5e00640 1 --2- 172.21.15.12:0/51153116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03e0104bf0 0x7f03e019a1d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f03c800e9f0 tx=0x7f03c800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.601+0000 7f03d7e00640 1 -- 172.21.15.12:0/51153116 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c800cd80 con 0x7f03e0104bf0 2024-03-17T23:55:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.601+0000 7f03d7e00640 1 -- 172.21.15.12:0/51153116 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03c8005500 con 0x7f03e0104bf0 2024-03-17T23:55:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.601+0000 7f03d7e00640 1 -- 172.21.15.12:0/51153116 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c8010690 con 0x7f03e0104bf0 2024-03-17T23:55:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.601+0000 7f03e5400640 1 -- 172.21.15.12:0/51153116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e01a1bc0 con 0x7f03e0104bf0 2024-03-17T23:55:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.601+0000 7f03e5400640 1 -- 172.21.15.12:0/51153116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e01a2110 con 0x7f03e0104bf0 2024-03-17T23:55:03.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.602+0000 7f03e5400640 1 -- 172.21.15.12:0/51153116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03e010acc0 con 0x7f03e0104bf0 2024-03-17T23:55:03.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.604+0000 7f03d7e00640 1 -- 172.21.15.12:0/51153116 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f03c80046c0 con 0x7f03e0104bf0 2024-03-17T23:55:03.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.605+0000 7f03d7e00640 1 --2- 172.21.15.12:0/51153116 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f03c40a2880 0x7f03c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:03.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.605+0000 7f03d7e00640 1 -- 172.21.15.12:0/51153116 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f03c80cf110 con 0x7f03e0104bf0 2024-03-17T23:55:03.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.605+0000 7f03de800640 1 --2- 172.21.15.12:0/51153116 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f03c40a2880 0x7f03c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:03.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.606+0000 7f03de800640 1 --2- 172.21.15.12:0/51153116 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f03c40a2880 0x7f03c40a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03e019ab60 tx=0x7f03d0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:03.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.609+0000 7f03d7e00640 1 -- 172.21.15.12:0/51153116 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f03c809bf10 con 0x7f03e0104bf0 2024-03-17T23:55:03.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.812+0000 7f03e5400640 1 -- 172.21.15.12:0/51153116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f03e00623e0 con 0x7f03e0104bf0 2024-03-17T23:55:03.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.814+0000 7f03d7e00640 1 -- 172.21.15.12:0/51153116 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24044 (secure 0 0 0) 0x7f03c809b810 con 0x7f03e0104bf0 2024-03-17T23:55:03.815 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:03.815 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":90,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:55:02.337441+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:55:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.817+0000 7f03e5400640 1 -- 172.21.15.12:0/51153116 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f03c40a2880 msgr2=0x7f03c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.817+0000 7f03e5400640 1 --2- 172.21.15.12:0/51153116 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f03c40a2880 0x7f03c40a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03e019ab60 tx=0x7f03d0009500 comp rx=0 tx=0).stop 2024-03-17T23:55:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.817+0000 7f03e5400640 1 -- 172.21.15.12:0/51153116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03e0104bf0 msgr2=0x7f03e019a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.817+0000 7f03e5400640 1 --2- 172.21.15.12:0/51153116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03e0104bf0 0x7f03e019a1d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f03c800e9f0 tx=0x7f03c800edf0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.817+0000 7f03e5400640 1 -- 172.21.15.12:0/51153116 shutdown_connections 2024-03-17T23:55:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.817+0000 7f03e5400640 1 --2- 172.21.15.12:0/51153116 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f03c40a2880 0x7f03c40a4d40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.817+0000 7f03e5400640 1 --2- 172.21.15.12:0/51153116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03e01055b0 0x7f03e01a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.817+0000 7f03e5400640 1 --2- 172.21.15.12:0/51153116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03e0104bf0 0x7f03e019a1d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.817+0000 7f03e5400640 1 --2- 172.21.15.12:0/51153116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03e0103a00 0x7f03e0199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:03.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.817+0000 7f03e5400640 1 -- 172.21.15.12:0/51153116 >> 172.21.15.12:0/51153116 conn(0x7f03e00fef90 msgr2=0x7f03e0107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:03.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.817+0000 7f03e5400640 1 -- 172.21.15.12:0/51153116 shutdown_connections 2024-03-17T23:55:03.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:03.817+0000 7f03e5400640 1 -- 172.21.15.12:0/51153116 wait complete. 2024-03-17T23:55:03.838 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:55:04.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.052+0000 7f9a6b200640 1 -- 172.21.15.12:0/1348884334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a6c108e10 msgr2=0x7f9a6c109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:04.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.052+0000 7f9a6b200640 1 --2- 172.21.15.12:0/1348884334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a6c108e10 0x7f9a6c109290 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9a54009900 tx=0x7f9a5402bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:04.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.052+0000 7f9a6b200640 1 -- 172.21.15.12:0/1348884334 shutdown_connections 2024-03-17T23:55:04.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.052+0000 7f9a6b200640 1 --2- 172.21.15.12:0/1348884334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a6c1097d0 0x7f9a6c110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.052+0000 7f9a6b200640 1 --2- 172.21.15.12:0/1348884334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a6c108e10 0x7f9a6c109290 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.052+0000 7f9a6b200640 1 --2- 172.21.15.12:0/1348884334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a6c107c20 0x7f9a6c108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.052+0000 7f9a6b200640 1 -- 172.21.15.12:0/1348884334 >> 172.21.15.12:0/1348884334 conn(0x7f9a6c076bb0 msgr2=0x7f9a6c078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:04.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.053+0000 7f9a6b200640 1 -- 172.21.15.12:0/1348884334 shutdown_connections 2024-03-17T23:55:04.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.053+0000 7f9a6b200640 1 -- 172.21.15.12:0/1348884334 wait complete. 2024-03-17T23:55:04.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.054+0000 7f9a6b200640 1 Processor -- start 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.054+0000 7f9a6b200640 1 -- start start 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.054+0000 7f9a6b200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a6c107c20 0x7f9a6c19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.054+0000 7f9a6b200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a6c108e10 0x7f9a6c19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a6b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a6c1097d0 0x7f9a6c1a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a6b200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c1138f0 con 0x7f9a6c108e10 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a6b200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c113770 con 0x7f9a6c107c20 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a6b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c113a70 con 0x7f9a6c1097d0 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a69e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a6c107c20 0x7f9a6c19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a69e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a6c107c20 0x7f9a6c19dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60542/0 (socket says 172.21.15.12:60542) 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a69e00640 1 -- 172.21.15.12:0/3187390298 learned_addr learned my addr 172.21.15.12:0/3187390298 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a63e00640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a6c108e10 0x7f9a6c19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a6a800640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a6c1097d0 0x7f9a6c1a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a69e00640 1 -- 172.21.15.12:0/3187390298 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a6c1097d0 msgr2=0x7f9a6c1a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:04.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a69e00640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a6c1097d0 0x7f9a6c1a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a69e00640 1 -- 172.21.15.12:0/3187390298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a6c108e10 msgr2=0x7f9a6c19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:04.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a69e00640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a6c108e10 0x7f9a6c19e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a69e00640 1 -- 172.21.15.12:0/3187390298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6c1a5b00 con 0x7f9a6c107c20 2024-03-17T23:55:04.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.055+0000 7f9a69e00640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a6c107c20 0x7f9a6c19dee0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00d950 tx=0x7f9a5c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:04.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.056+0000 7f9a62a00640 1 -- 172.21.15.12:0/3187390298 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c017020 con 0x7f9a6c107c20 2024-03-17T23:55:04.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.056+0000 7f9a62a00640 1 -- 172.21.15.12:0/3187390298 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a5c009c30 con 0x7f9a6c107c20 2024-03-17T23:55:04.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.056+0000 7f9a6b200640 1 -- 172.21.15.12:0/3187390298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a6c1a5df0 con 0x7f9a6c107c20 2024-03-17T23:55:04.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.056+0000 7f9a62a00640 1 -- 172.21.15.12:0/3187390298 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c00f720 con 0x7f9a6c107c20 2024-03-17T23:55:04.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.056+0000 7f9a6b200640 1 -- 172.21.15.12:0/3187390298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a6c1a6250 con 0x7f9a6c107c20 2024-03-17T23:55:04.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.057+0000 7f9a6b200640 1 -- 172.21.15.12:0/3187390298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a6c10ef40 con 0x7f9a6c107c20 2024-03-17T23:55:04.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.058+0000 7f9a62a00640 1 -- 172.21.15.12:0/3187390298 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9a5c01a020 con 0x7f9a6c107c20 2024-03-17T23:55:04.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.060+0000 7f9a62a00640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9a500a2800 0x7f9a500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:04.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.060+0000 7f9a62a00640 1 -- 172.21.15.12:0/3187390298 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f9a5c0cec80 con 0x7f9a6c107c20 2024-03-17T23:55:04.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.060+0000 7f9a63e00640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9a500a2800 0x7f9a500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:04.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.061+0000 7f9a63e00640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9a500a2800 0x7f9a500a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9a5402c240 tx=0x7f9a5402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.064+0000 7f9a62a00640 1 -- 172.21.15.12:0/3187390298 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9a5c021020 con 0x7f9a6c107c20 2024-03-17T23:55:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:03 smithi012 ceph-mon[31541]: pgmap v323: 129 pgs: 2 creating+peering, 30 unknown, 97 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-03-17T23:55:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:03 smithi012 ceph-mon[31541]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-17T23:55:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:03 smithi012 ceph-mon[31541]: osdmap e90: 12 total, 12 up, 12 in 2024-03-17T23:55:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:03 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2319885377' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-17T23:55:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:03 smithi012 ceph-mon[31541]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:55:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:03 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3997013588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:55:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:03 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/51153116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:04.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.268+0000 7f9a6b200640 1 -- 172.21.15.12:0/3187390298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9a6c062480 con 0x7f9a6c107c20 2024-03-17T23:55:04.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.270+0000 7f9a62a00640 1 -- 172.21.15.12:0/3187390298 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24044 (secure 0 0 0) 0x7f9a5c009de0 con 0x7f9a6c107c20 2024-03-17T23:55:04.271 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:04.271 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":90,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:55:02.337441+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:55:04.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.273+0000 7f9a6b200640 1 -- 172.21.15.12:0/3187390298 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9a500a2800 msgr2=0x7f9a500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:04.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.273+0000 7f9a6b200640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9a500a2800 0x7f9a500a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9a5402c240 tx=0x7f9a5402df90 comp rx=0 tx=0).stop 2024-03-17T23:55:04.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.273+0000 7f9a6b200640 1 -- 172.21.15.12:0/3187390298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a6c107c20 msgr2=0x7f9a6c19dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:04.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.273+0000 7f9a6b200640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a6c107c20 0x7f9a6c19dee0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00d950 tx=0x7f9a5c00de10 comp rx=0 tx=0).stop 2024-03-17T23:55:04.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.273+0000 7f9a6b200640 1 -- 172.21.15.12:0/3187390298 shutdown_connections 2024-03-17T23:55:04.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.273+0000 7f9a6b200640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9a500a2800 0x7f9a500a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.273+0000 7f9a6b200640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a6c1097d0 0x7f9a6c1a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.273+0000 7f9a6b200640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a6c108e10 0x7f9a6c19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.273+0000 7f9a6b200640 1 --2- 172.21.15.12:0/3187390298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a6c107c20 0x7f9a6c19dee0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.273+0000 7f9a6b200640 1 -- 172.21.15.12:0/3187390298 >> 172.21.15.12:0/3187390298 conn(0x7f9a6c076bb0 msgr2=0x7f9a6c10c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:04.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.273+0000 7f9a6b200640 1 -- 172.21.15.12:0/3187390298 shutdown_connections 2024-03-17T23:55:04.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.273+0000 7f9a6b200640 1 -- 172.21.15.12:0/3187390298 wait complete. 2024-03-17T23:55:04.312 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:55:04.313 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:03 smithi079 ceph-mon[36241]: pgmap v323: 129 pgs: 2 creating+peering, 30 unknown, 97 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-03-17T23:55:04.313 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:03 smithi079 ceph-mon[36241]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-17T23:55:04.313 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:03 smithi079 ceph-mon[36241]: osdmap e90: 12 total, 12 up, 12 in 2024-03-17T23:55:04.314 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:03 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2319885377' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-17T23:55:04.314 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:03 smithi079 ceph-mon[36241]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:55:04.314 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:03 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3997013588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:55:04.314 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:03 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/51153116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:03 smithi191 ceph-mon[36708]: pgmap v323: 129 pgs: 2 creating+peering, 30 unknown, 97 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-03-17T23:55:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:03 smithi191 ceph-mon[36708]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-17T23:55:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:03 smithi191 ceph-mon[36708]: osdmap e90: 12 total, 12 up, 12 in 2024-03-17T23:55:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:03 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2319885377' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-17T23:55:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:03 smithi191 ceph-mon[36708]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:55:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:03 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3997013588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:55:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:03 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/51153116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:04.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.534+0000 7ff8eae00640 1 -- 172.21.15.12:0/3974662984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e4108e10 msgr2=0x7ff8e4109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:04.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.534+0000 7ff8eae00640 1 --2- 172.21.15.12:0/3974662984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e4108e10 0x7ff8e4109290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0009900 tx=0x7ff8d002bc40 comp rx=0 tx=0).stop 2024-03-17T23:55:04.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.534+0000 7ff8eae00640 1 -- 172.21.15.12:0/3974662984 shutdown_connections 2024-03-17T23:55:04.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.534+0000 7ff8eae00640 1 --2- 172.21.15.12:0/3974662984 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff8e41097d0 0x7ff8e4110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.534+0000 7ff8eae00640 1 --2- 172.21.15.12:0/3974662984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e4108e10 0x7ff8e4109290 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.534+0000 7ff8eae00640 1 --2- 172.21.15.12:0/3974662984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8e4107c20 0x7ff8e4108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.534+0000 7ff8eae00640 1 -- 172.21.15.12:0/3974662984 >> 172.21.15.12:0/3974662984 conn(0x7ff8e4076bb0 msgr2=0x7ff8e4078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:04.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.534+0000 7ff8eae00640 1 -- 172.21.15.12:0/3974662984 shutdown_connections 2024-03-17T23:55:04.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.534+0000 7ff8eae00640 1 -- 172.21.15.12:0/3974662984 wait complete. 2024-03-17T23:55:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.535+0000 7ff8eae00640 1 Processor -- start 2024-03-17T23:55:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.536+0000 7ff8eae00640 1 -- start start 2024-03-17T23:55:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.536+0000 7ff8eae00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff8e4107c20 0x7ff8e419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.536+0000 7ff8eae00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e4108e10 0x7ff8e419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.536+0000 7ff8eae00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8e41097d0 0x7ff8e41a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.536+0000 7ff8eae00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e41138a0 con 0x7ff8e41097d0 2024-03-17T23:55:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.536+0000 7ff8eae00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4113720 con 0x7ff8e4107c20 2024-03-17T23:55:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.536+0000 7ff8eae00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4113a20 con 0x7ff8e4108e10 2024-03-17T23:55:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.536+0000 7ff8e9a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff8e4107c20 0x7ff8e419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.536+0000 7ff8e9a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff8e4107c20 0x7ff8e419dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60574/0 (socket says 172.21.15.12:60574) 2024-03-17T23:55:04.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.536+0000 7ff8e9a00640 1 -- 172.21.15.12:0/4081761450 learned_addr learned my addr 172.21.15.12:0/4081761450 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:04.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.536+0000 7ff8e9000640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e4108e10 0x7ff8e419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:04.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.536+0000 7ff8ea400640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8e41097d0 0x7ff8e41a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:04.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.537+0000 7ff8e9a00640 1 -- 172.21.15.12:0/4081761450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e4108e10 msgr2=0x7ff8e419e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:04.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.537+0000 7ff8e9a00640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e4108e10 0x7ff8e419e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.537+0000 7ff8e9a00640 1 -- 172.21.15.12:0/4081761450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8e41097d0 msgr2=0x7ff8e41a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:04.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.537+0000 7ff8e9a00640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8e41097d0 0x7ff8e41a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.537+0000 7ff8e9a00640 1 -- 172.21.15.12:0/4081761450 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e41a5b00 con 0x7ff8e4107c20 2024-03-17T23:55:04.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.537+0000 7ff8e9a00640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff8e4107c20 0x7ff8e419dee0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800e9c0 tx=0x7ff8d800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:04.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.537+0000 7ff8e2000640 1 -- 172.21.15.12:0/4081761450 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d800cd80 con 0x7ff8e4107c20 2024-03-17T23:55:04.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.537+0000 7ff8e2000640 1 -- 172.21.15.12:0/4081761450 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8d80035f0 con 0x7ff8e4107c20 2024-03-17T23:55:04.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.537+0000 7ff8e2000640 1 -- 172.21.15.12:0/4081761450 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8010690 con 0x7ff8e4107c20 2024-03-17T23:55:04.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.537+0000 7ff8eae00640 1 -- 172.21.15.12:0/4081761450 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e41a5df0 con 0x7ff8e4107c20 2024-03-17T23:55:04.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.538+0000 7ff8eae00640 1 -- 172.21.15.12:0/4081761450 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e41a6260 con 0x7ff8e4107c20 2024-03-17T23:55:04.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.539+0000 7ff8eae00640 1 -- 172.21.15.12:0/4081761450 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e410ef40 con 0x7ff8e4107c20 2024-03-17T23:55:04.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.541+0000 7ff8e2000640 1 -- 172.21.15.12:0/4081761450 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff8d80037a0 con 0x7ff8e4107c20 2024-03-17T23:55:04.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.542+0000 7ff8e2000640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff8cc0a2880 0x7ff8cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:04.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.543+0000 7ff8e2000640 1 -- 172.21.15.12:0/4081761450 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff8d80ceff0 con 0x7ff8e4107c20 2024-03-17T23:55:04.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.543+0000 7ff8e9000640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff8cc0a2880 0x7ff8cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:04.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.543+0000 7ff8e9000640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff8cc0a2880 0x7ff8cc0a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff8d002c7c0 tx=0x7ff8d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:04.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.547+0000 7ff8e2000640 1 -- 172.21.15.12:0/4081761450 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff8d809be10 con 0x7ff8e4107c20 2024-03-17T23:55:04.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.795+0000 7ff8eae00640 1 -- 172.21.15.12:0/4081761450 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8e402d1b0 con 0x7ff8e4107c20 2024-03-17T23:55:04.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.796+0000 7ff8e2000640 1 -- 172.21.15.12:0/4081761450 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11800 (secure 0 0 0) 0x7ff8e402d1b0 con 0x7ff8e4107c20 2024-03-17T23:55:04.797 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:04.797 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/2073607248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":2073607248},{"type":"v1","addr":"172.21.15.12:6841","nonce":2073607248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:54:56.312554+0000","modified":"2024-03-17T23:55:02.337827+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/2478742711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":2478742711},{"type":"v1","addr":"172.21.15.79:6839","nonce":2478742711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-03-17T23:55:04.797 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 17 2024-03-17T23:55:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.798+0000 7ff8eae00640 1 -- 172.21.15.12:0/4081761450 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff8cc0a2880 msgr2=0x7ff8cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.798+0000 7ff8eae00640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff8cc0a2880 0x7ff8cc0a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff8d002c7c0 tx=0x7ff8d0005fd0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.798+0000 7ff8eae00640 1 -- 172.21.15.12:0/4081761450 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff8e4107c20 msgr2=0x7ff8e419dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.799+0000 7ff8eae00640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff8e4107c20 0x7ff8e419dee0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800e9c0 tx=0x7ff8d800ee80 comp rx=0 tx=0).stop 2024-03-17T23:55:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.799+0000 7ff8eae00640 1 -- 172.21.15.12:0/4081761450 shutdown_connections 2024-03-17T23:55:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.799+0000 7ff8eae00640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff8cc0a2880 0x7ff8cc0a4d40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.799+0000 7ff8eae00640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8e41097d0 0x7ff8e41a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.799+0000 7ff8eae00640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e4108e10 0x7ff8e419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.799+0000 7ff8eae00640 1 --2- 172.21.15.12:0/4081761450 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff8e4107c20 0x7ff8e419dee0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.799+0000 7ff8eae00640 1 -- 172.21.15.12:0/4081761450 >> 172.21.15.12:0/4081761450 conn(0x7ff8e4076bb0 msgr2=0x7ff8e410c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.799+0000 7ff8eae00640 1 -- 172.21.15.12:0/4081761450 shutdown_connections 2024-03-17T23:55:04.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:04.799+0000 7ff8eae00640 1 -- 172.21.15.12:0/4081761450 wait complete. 2024-03-17T23:55:04.820 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:55:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.045+0000 7f7a80a00640 1 -- 172.21.15.12:0/417319284 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a7c1055b0 msgr2=0x7f7a7c10be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.045+0000 7f7a80a00640 1 --2- 172.21.15.12:0/417319284 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a7c1055b0 0x7f7a7c10be40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009900 tx=0x7f7a7402bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.045+0000 7f7a80a00640 1 -- 172.21.15.12:0/417319284 shutdown_connections 2024-03-17T23:55:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.045+0000 7f7a80a00640 1 --2- 172.21.15.12:0/417319284 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a7c1055b0 0x7f7a7c10be40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.045+0000 7f7a80a00640 1 --2- 172.21.15.12:0/417319284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c104bf0 0x7f7a7c105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.045+0000 7f7a80a00640 1 --2- 172.21.15.12:0/417319284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a7c103a00 0x7f7a7c103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.045+0000 7f7a80a00640 1 -- 172.21.15.12:0/417319284 >> 172.21.15.12:0/417319284 conn(0x7f7a7c0fef90 msgr2=0x7f7a7c1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.045+0000 7f7a80a00640 1 -- 172.21.15.12:0/417319284 shutdown_connections 2024-03-17T23:55:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.045+0000 7f7a80a00640 1 -- 172.21.15.12:0/417319284 wait complete. 2024-03-17T23:55:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.046+0000 7f7a80a00640 1 Processor -- start 2024-03-17T23:55:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.046+0000 7f7a80a00640 1 -- start start 2024-03-17T23:55:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.046+0000 7f7a80a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a7c103a00 0x7f7a7c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.046+0000 7f7a80a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c104bf0 0x7f7a7c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.046+0000 7f7a80a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a7c1055b0 0x7f7a7c1a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.046+0000 7f7a80a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7c10f6d0 con 0x7f7a7c104bf0 2024-03-17T23:55:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.046+0000 7f7a80a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7c10f550 con 0x7f7a7c1055b0 2024-03-17T23:55:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.046+0000 7f7a80a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7c10f850 con 0x7f7a7c103a00 2024-03-17T23:55:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.047+0000 7f7a79400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c104bf0 0x7f7a7c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.047+0000 7f7a79400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c104bf0 0x7f7a7c19a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41980/0 (socket says 172.21.15.12:41980) 2024-03-17T23:55:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.047+0000 7f7a79400640 1 -- 172.21.15.12:0/3674217268 learned_addr learned my addr 172.21.15.12:0/3674217268 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.047+0000 7f7a79e00640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a7c103a00 0x7f7a7c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.047+0000 7f7a7a800640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a7c1055b0 0x7f7a7c1a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.047+0000 7f7a79400640 1 -- 172.21.15.12:0/3674217268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a7c103a00 msgr2=0x7f7a7c199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.047+0000 7f7a79400640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a7c103a00 0x7f7a7c199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.047+0000 7f7a79400640 1 -- 172.21.15.12:0/3674217268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a7c1055b0 msgr2=0x7f7a7c1a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.047+0000 7f7a79400640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a7c1055b0 0x7f7a7c1a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.047+0000 7f7a79400640 1 -- 172.21.15.12:0/3674217268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a7c1a1880 con 0x7f7a7c104bf0 2024-03-17T23:55:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.047+0000 7f7a79e00640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a7c103a00 0x7f7a7c199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:55:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.047+0000 7f7a79400640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c104bf0 0x7f7a7c19a1b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7a6400ebc0 tx=0x7f7a6400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.048+0000 7f7a72a00640 1 -- 172.21.15.12:0/3674217268 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a64010630 con 0x7f7a7c104bf0 2024-03-17T23:55:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.048+0000 7f7a80a00640 1 -- 172.21.15.12:0/3674217268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a7c1a1b70 con 0x7f7a7c104bf0 2024-03-17T23:55:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.048+0000 7f7a72a00640 1 -- 172.21.15.12:0/3674217268 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a64010cb0 con 0x7f7a7c104bf0 2024-03-17T23:55:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.048+0000 7f7a72a00640 1 -- 172.21.15.12:0/3674217268 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a64018bd0 con 0x7f7a7c104bf0 2024-03-17T23:55:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.048+0000 7f7a80a00640 1 -- 172.21.15.12:0/3674217268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a7c1a20c0 con 0x7f7a7c104bf0 2024-03-17T23:55:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.049+0000 7f7a80a00640 1 -- 172.21.15.12:0/3674217268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a7c10ac50 con 0x7f7a7c104bf0 2024-03-17T23:55:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.051+0000 7f7a72a00640 1 -- 172.21.15.12:0/3674217268 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7a640107d0 con 0x7f7a7c104bf0 2024-03-17T23:55:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.053+0000 7f7a72a00640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7a600a2800 0x7f7a600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.053+0000 7f7a72a00640 1 -- 172.21.15.12:0/3674217268 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f7a640cf5b0 con 0x7f7a7c104bf0 2024-03-17T23:55:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.053+0000 7f7a79e00640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7a600a2800 0x7f7a600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.054+0000 7f7a79e00640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7a600a2800 0x7f7a600a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c19b140 tx=0x7f7a6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.055+0000 7f7a72a00640 1 -- 172.21.15.12:0/3674217268 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7a6409bd60 con 0x7f7a7c104bf0 2024-03-17T23:55:05.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:04 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3187390298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:05.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:04 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4081761450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:55:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.261+0000 7f7a80a00640 1 -- 172.21.15.12:0/3674217268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7a7c02d380 con 0x7f7a7c104bf0 2024-03-17T23:55:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.263+0000 7f7a72a00640 1 -- 172.21.15.12:0/3674217268 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24044 (secure 0 0 0) 0x7f7a6409fe00 con 0x7f7a7c104bf0 2024-03-17T23:55:05.264 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:05.264 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":90,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:55:02.337441+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:55:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.266+0000 7f7a80a00640 1 -- 172.21.15.12:0/3674217268 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7a600a2800 msgr2=0x7f7a600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.266+0000 7f7a80a00640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7a600a2800 0x7f7a600a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c19b140 tx=0x7f7a6c005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.266+0000 7f7a80a00640 1 -- 172.21.15.12:0/3674217268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c104bf0 msgr2=0x7f7a7c19a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.266+0000 7f7a80a00640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c104bf0 0x7f7a7c19a1b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7a6400ebc0 tx=0x7f7a6400c6a0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.266+0000 7f7a80a00640 1 -- 172.21.15.12:0/3674217268 shutdown_connections 2024-03-17T23:55:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.266+0000 7f7a80a00640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7a600a2800 0x7f7a600a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.266+0000 7f7a80a00640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a7c1055b0 0x7f7a7c1a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.266+0000 7f7a80a00640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a7c104bf0 0x7f7a7c19a1b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.266+0000 7f7a80a00640 1 --2- 172.21.15.12:0/3674217268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a7c103a00 0x7f7a7c199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.266+0000 7f7a80a00640 1 -- 172.21.15.12:0/3674217268 >> 172.21.15.12:0/3674217268 conn(0x7f7a7c0fef90 msgr2=0x7f7a7c107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.266+0000 7f7a80a00640 1 -- 172.21.15.12:0/3674217268 shutdown_connections 2024-03-17T23:55:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.266+0000 7f7a80a00640 1 -- 172.21.15.12:0/3674217268 wait complete. 2024-03-17T23:55:05.288 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-17T23:54:56.312554+0000', 'modified': '2024-03-17T23:55:02.337827+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24587}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.79:6839/2478742711', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6838', 'nonce': 2478742711}, {'type': 'v1', 'addr': '172.21.15.79:6839', 'nonce': 2478742711}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24587, 'qdb_cluster': [24587]} 2024-03-17T23:55:05.288 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-17T23:55:05.288 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-17T23:55:05.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:04 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3187390298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:05.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:04 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/4081761450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:55:05.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:04 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3187390298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:05.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:04 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/4081761450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:55:05.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.496+0000 7f3ee7e00640 1 -- 172.21.15.12:0/3431947108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ee8102170 msgr2=0x7f3ee8108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.496+0000 7f3ee7e00640 1 --2- 172.21.15.12:0/3431947108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ee8102170 0x7f3ee8108a00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0009be0 tx=0x7f3ee002bef0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.496+0000 7f3ee7e00640 1 -- 172.21.15.12:0/3431947108 shutdown_connections 2024-03-17T23:55:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.496+0000 7f3ee7e00640 1 --2- 172.21.15.12:0/3431947108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ee8102170 0x7f3ee8108a00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.496+0000 7f3ee7e00640 1 --2- 172.21.15.12:0/3431947108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ee81017b0 0x7f3ee8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.496+0000 7f3ee7e00640 1 --2- 172.21.15.12:0/3431947108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee81005c0 0x7f3ee81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.496+0000 7f3ee7e00640 1 -- 172.21.15.12:0/3431947108 >> 172.21.15.12:0/3431947108 conn(0x7f3ee80fbb50 msgr2=0x7f3ee80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.496+0000 7f3ee7e00640 1 -- 172.21.15.12:0/3431947108 shutdown_connections 2024-03-17T23:55:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.496+0000 7f3ee7e00640 1 -- 172.21.15.12:0/3431947108 wait complete. 2024-03-17T23:55:05.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.497+0000 7f3ee7e00640 1 Processor -- start 2024-03-17T23:55:05.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.497+0000 7f3ee7e00640 1 -- start start 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.497+0000 7f3ee7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ee81005c0 0x7f3ee81045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.497+0000 7f3ee7e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ee81017b0 0x7f3ee8102c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.497+0000 7f3ee7e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee8102170 0x7f3ee8103140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.497+0000 7f3ee7e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee810c270 con 0x7f3ee8102170 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.497+0000 7f3ee7e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee810c0f0 con 0x7f3ee81017b0 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.497+0000 7f3ee7e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee810c3f0 con 0x7f3ee81005c0 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.498+0000 7f3edfe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ee81017b0 0x7f3ee8102c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.498+0000 7f3ee6a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ee81005c0 0x7f3ee81045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.498+0000 7f3edfe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ee81017b0 0x7f3ee8102c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60616/0 (socket says 172.21.15.12:60616) 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.498+0000 7f3edfe00640 1 -- 172.21.15.12:0/2688241340 learned_addr learned my addr 172.21.15.12:0/2688241340 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.498+0000 7f3ee7400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee8102170 0x7f3ee8103140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.498+0000 7f3edfe00640 1 -- 172.21.15.12:0/2688241340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ee81005c0 msgr2=0x7f3ee81045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.498+0000 7f3edfe00640 1 --2- 172.21.15.12:0/2688241340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ee81005c0 0x7f3ee81045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.498+0000 7f3edfe00640 1 -- 172.21.15.12:0/2688241340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee8102170 msgr2=0x7f3ee8103140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.498+0000 7f3edfe00640 1 --2- 172.21.15.12:0/2688241340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee8102170 0x7f3ee8103140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.498+0000 7f3edfe00640 1 -- 172.21.15.12:0/2688241340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee81039d0 con 0x7f3ee81017b0 2024-03-17T23:55:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.498+0000 7f3edfe00640 1 --2- 172.21.15.12:0/2688241340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ee81017b0 0x7f3ee8102c00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000d950 tx=0x7f3ed000dd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.499+0000 7f3edf400640 1 -- 172.21.15.12:0/2688241340 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0017070 con 0x7f3ee81017b0 2024-03-17T23:55:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.499+0000 7f3edf400640 1 -- 172.21.15.12:0/2688241340 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ed000de80 con 0x7f3ee81017b0 2024-03-17T23:55:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.499+0000 7f3edf400640 1 -- 172.21.15.12:0/2688241340 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed000f620 con 0x7f3ee81017b0 2024-03-17T23:55:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.499+0000 7f3ee7e00640 1 -- 172.21.15.12:0/2688241340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ee81a1ad0 con 0x7f3ee81017b0 2024-03-17T23:55:05.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.499+0000 7f3ee7e00640 1 -- 172.21.15.12:0/2688241340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ee81a1fd0 con 0x7f3ee81017b0 2024-03-17T23:55:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.502+0000 7f3edf400640 1 -- 172.21.15.12:0/2688241340 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3ed0009c30 con 0x7f3ee81017b0 2024-03-17T23:55:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.502+0000 7f3ef039d740 1 Processor -- start 2024-03-17T23:55:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.502+0000 7f3ef039d740 1 -- start start 2024-03-17T23:55:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.502+0000 7f3ef039d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cf340 0x55f5ee9cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.502+0000 7f3ef039d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.502+0000 7f3ef039d740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9d7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.502+0000 7f3ef039d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f5ee933c50 con 0x55f5ee9cfd30 2024-03-17T23:55:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.502+0000 7f3ef039d740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f5ee8dc9d0 con 0x55f5ee9d4ea0 2024-03-17T23:55:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.502+0000 7f3ef039d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f5ee92fcf0 con 0x55f5ee9cf340 2024-03-17T23:55:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.503+0000 7f3edfe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9d7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.503+0000 7f3ee6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.503+0000 7f3edfe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9d7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60632/0 (socket says 172.21.15.12:60632) 2024-03-17T23:55:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.503+0000 7f3edfe00640 1 -- 172.21.15.12:0/3055845768 learned_addr learned my addr 172.21.15.12:0/3055845768 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.503+0000 7f3ee7400640 1 --2- 172.21.15.12:0/3055845768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cf340 0x55f5ee9cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.503+0000 7f3edfe00640 1 -- 172.21.15.12:0/3055845768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cf340 msgr2=0x55f5ee9cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:55:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.503+0000 7f3edfe00640 1 --2- 172.21.15.12:0/3055845768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cf340 0x55f5ee9cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.503+0000 7f3edfe00640 1 -- 172.21.15.12:0/3055845768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9cfd30 msgr2=0x55f5ee9d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:55:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.503+0000 7f3edfe00640 1 --2- 172.21.15.12:0/3055845768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.503+0000 7f3edfe00640 1 -- 172.21.15.12:0/3055845768 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5ee8f6f10 con 0x55f5ee9d4ea0 2024-03-17T23:55:05.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.503+0000 7f3edf400640 1 --2- 172.21.15.12:0/2688241340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ecc0a2880 0x7f3ecc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.504+0000 7f3edfe00640 1 --2- 172.21.15.12:0/3055845768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9d7330 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3ed00d4c20 tx=0x7f3ed00d6670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9393719d1bc4914d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:05.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.504+0000 7f3ed7400640 1 -- 172.21.15.12:0/3055845768 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0097210 con 0x55f5ee9d4ea0 2024-03-17T23:55:05.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.504+0000 7f3ee7400640 1 --2- 172.21.15.12:0/2688241340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ecc0a2880 0x7f3ecc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.504+0000 7f3ed7400640 1 -- 172.21.15.12:0/3055845768 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ed009b670 con 0x55f5ee9d4ea0 2024-03-17T23:55:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.504+0000 7f3ed7400640 1 -- 172.21.15.12:0/3055845768 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed006fe50 con 0x55f5ee9d4ea0 2024-03-17T23:55:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.505+0000 7f3edf400640 1 -- 172.21.15.12:0/2688241340 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f3ed00ce870 con 0x7f3ee81017b0 2024-03-17T23:55:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.505+0000 7f3ef039d740 1 -- 172.21.15.12:0/3055845768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 msgr2=0x55f5ee9d7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.505+0000 7f3ee7400640 1 --2- 172.21.15.12:0/2688241340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ecc0a2880 0x7f3ecc0a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0006010 tx=0x7f3ee00333e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.505+0000 7f3ef039d740 1 --2- 172.21.15.12:0/3055845768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9d7330 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3ed00d4c20 tx=0x7f3ed00d6670 comp rx=0 tx=0).stop 2024-03-17T23:55:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.505+0000 7f3ef039d740 1 -- 172.21.15.12:0/3055845768 shutdown_connections 2024-03-17T23:55:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.505+0000 7f3ef039d740 1 --2- 172.21.15.12:0/3055845768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9d7330 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.505+0000 7f3ef039d740 1 --2- 172.21.15.12:0/3055845768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.505+0000 7f3ef039d740 1 --2- 172.21.15.12:0/3055845768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cf340 0x55f5ee9cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.505+0000 7f3ef039d740 1 -- 172.21.15.12:0/3055845768 >> 172.21.15.12:0/3055845768 conn(0x55f5ee9ca8b0 msgr2=0x55f5ee9cacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.505+0000 7f3ef039d740 1 -- 172.21.15.12:0/3055845768 shutdown_connections 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.505+0000 7f3ef039d740 1 -- 172.21.15.12:0/3055845768 wait complete. 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.506+0000 7f3ef039d740 1 -- start start 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.506+0000 7f3ef039d740 1 Processor -- start 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.506+0000 7f3ef039d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cf340 0x55f5ee9d9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.506+0000 7f3ef039d740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9d9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.506+0000 7f3ef039d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9e09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.506+0000 7f3ef039d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f5ee9d7e50 con 0x55f5ee9d4ea0 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.506+0000 7f3ef039d740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f5ee8efe20 con 0x55f5ee9cfd30 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.506+0000 7f3ef039d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f5ee9d7fd0 con 0x55f5ee9cf340 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3ef039d740 10 client.0 _ll_register_callbacks cb 0x55f5ee8ef7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3edfe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9d9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3edfe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9d9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60642/0 (socket says 172.21.15.12:60642) 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3edfe00640 1 -- 172.21.15.12:0/1185229601 learned_addr learned my addr 172.21.15.12:0/1185229601 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3ee7400640 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9e09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3ee6a00640 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cf340 0x55f5ee9d9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3edfe00640 1 -- 172.21.15.12:0/1185229601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cf340 msgr2=0x55f5ee9d9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3edfe00640 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cf340 0x55f5ee9d9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3edfe00640 1 -- 172.21.15.12:0/1185229601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9d4ea0 msgr2=0x55f5ee9e09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3edfe00640 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9e09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3edfe00640 1 -- 172.21.15.12:0/1185229601 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5ee6482d0 con 0x55f5ee9cfd30 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3ee6a00640 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cf340 0x55f5ee9d9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3edfe00640 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9d9970 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3ed009b820 tx=0x7f3ed0005bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32069a64d6eb06cf server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3ecbe00640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3ecbe00640 1 -- 172.21.15.12:0/1185229601 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0020020 con 0x55f5ee9cfd30 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3ecbe00640 1 -- 172.21.15.12:0/1185229601 <== mon.2 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f3ed00d46d0 con 0x55f5ee9cfd30 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.507+0000 7f3ef039d740 1 -- 172.21.15.12:0/1185229601 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ed009b670 con 0x55f5ee9cfd30 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.508+0000 7f3ecbe00640 1 -- 172.21.15.12:0/1185229601 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f3ed0070040 con 0x55f5ee9cfd30 2024-03-17T23:55:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.508+0000 7f3ef039d740 10 client.24695 fetch_fsmap learned FSMap version 17 2024-03-17T23:55:05.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.508+0000 7f3ef039d740 1 -- 172.21.15.12:0/1185229601 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55f5ee7126b0 con 0x55f5ee9cfd30 2024-03-17T23:55:05.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.508+0000 7f3ecbe00640 1 -- 172.21.15.12:0/1185229601 <== mon.2 v2:172.21.15.79:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ed009b670 con 0x55f5ee9cfd30 2024-03-17T23:55:05.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.508+0000 7f3ecbe00640 1 -- 172.21.15.12:0/1185229601 <== mon.2 v2:172.21.15.79:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f3ed00d4d10 con 0x55f5ee9cfd30 2024-03-17T23:55:05.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.508+0000 7f3ef039d740 10 client.24695 fetch_fsmap finished waiting for FSMap version 17 2024-03-17T23:55:05.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.508+0000 7f3ef039d740 10 client.24695 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-17T23:55:05.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.508+0000 7f3ef039d740 20 client.24695 populate_metadata read hostname 'smithi012' 2024-03-17T23:55:05.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.508+0000 7f3ef039d740 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55f5ee9e5d10 0x55f5ee9e7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.508+0000 7f3ef039d740 4 client.24695 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:55:05.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.508+0000 7f3ef039d740 1 -- 172.21.15.12:0/1185229601 --> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5ee9912b0 con 0x55f5ee9e5d10 2024-03-17T23:55:05.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.508+0000 7f3ee6a00640 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55f5ee9e5d10 0x55f5ee9e7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.509+0000 7f3ee6a00640 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55f5ee9e5d10 0x55f5ee9e7180 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ac3b67182ef8079 server_cookie=326c08b7dbb16435 in_seq=0 out_seq=0 2024-03-17T23:55:05.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.509+0000 7f3ecbe00640 10 client.24695 ms_handle_connect on v2:172.21.15.79:6838/2478742711 2024-03-17T23:55:05.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ecbe00640 1 -- 172.21.15.12:0/1185229601 <== mds.0 v2:172.21.15.79:6838/2478742711 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f5ee9912b0 con 0x55f5ee9e5d10 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ecbe00640 10 client.24695 handle_command_reply: tid=0 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ef039d740 1 client.24695 shutdown 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ef039d740 1 -- 172.21.15.12:0/1185229601 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cfd30 msgr2=0x55f5ee9d9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ef039d740 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9d9970 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3ed009b820 tx=0x7f3ed0005bb0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ef039d740 1 -- 172.21.15.12:0/1185229601 shutdown_connections 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ef039d740 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55f5ee9e5d10 0x55f5ee9e7180 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ef039d740 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9e09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ef039d740 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9d9970 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ef039d740 1 --2- 172.21.15.12:0/1185229601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cf340 0x55f5ee9d9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ecbe00640 0 client.24695 ms_handle_reset on v2:172.21.15.79:6838/2478742711 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ef039d740 1 -- 172.21.15.12:0/1185229601 >> 172.21.15.12:0/1185229601 conn(0x55f5ee9ca8b0 msgr2=0x55f5ee9c9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ef039d740 1 -- 172.21.15.12:0/1185229601 shutdown_connections 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ef039d740 1 -- 172.21.15.12:0/1185229601 wait complete. 2024-03-17T23:55:05.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.510+0000 7f3ef039d740 20 client.24695 trim_cache size 0 max 16384 2024-03-17T23:55:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3ef039d740 1 Processor -- start 2024-03-17T23:55:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3ef039d740 1 -- start start 2024-03-17T23:55:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3ef039d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9cf340 0x55f5ee9cac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3ef039d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9e0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3ef039d740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3ef039d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f5ee9e54c0 con 0x55f5ee9cf340 2024-03-17T23:55:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3ef039d740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f5ee9e5b90 con 0x55f5ee9d4ea0 2024-03-17T23:55:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3ef039d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f5ee9e6260 con 0x55f5ee9cfd30 2024-03-17T23:55:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3ee7400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3ee6a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9cf340 0x55f5ee9cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3edfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9e0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3ee7400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9c98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60656/0 (socket says 172.21.15.12:60656) 2024-03-17T23:55:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3edfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9e0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:58124/0 (socket says 172.21.15.12:58124) 2024-03-17T23:55:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.531+0000 7f3ee7400640 1 -- 172.21.15.12:0/2787269523 learned_addr learned my addr 172.21.15.12:0/2787269523 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.532+0000 7f3edfe00640 1 -- 172.21.15.12:0/2787269523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 msgr2=0x55f5ee9c98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:55:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.532+0000 7f3edfe00640 1 --2- 172.21.15.12:0/2787269523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9c98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.532+0000 7f3edfe00640 1 -- 172.21.15.12:0/2787269523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9cf340 msgr2=0x55f5ee9cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:55:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.532+0000 7f3edfe00640 1 --2- 172.21.15.12:0/2787269523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9cf340 0x55f5ee9cac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.532+0000 7f3edfe00640 1 -- 172.21.15.12:0/2787269523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5ee9d7f10 con 0x55f5ee9cfd30 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.532+0000 7f3ee7400640 1 --2- 172.21.15.12:0/2787269523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9c98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.532+0000 7f3edfe00640 1 --2- 172.21.15.12:0/2787269523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9e0970 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0070400 tx=0x7f3ed00d4530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85fb9b427b303db3 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.532+0000 7f3ed6a00640 1 -- 172.21.15.12:0/2787269523 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0020070 con 0x55f5ee9cfd30 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.532+0000 7f3ed6a00640 1 -- 172.21.15.12:0/2787269523 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ed009b670 con 0x55f5ee9cfd30 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.532+0000 7f3ed6a00640 1 -- 172.21.15.12:0/2787269523 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0070240 con 0x55f5ee9cfd30 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.533+0000 7f3ef039d740 1 -- 172.21.15.12:0/2787269523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cfd30 msgr2=0x55f5ee9e0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.533+0000 7f3ef039d740 1 --2- 172.21.15.12:0/2787269523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9e0970 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0070400 tx=0x7f3ed00d4530 comp rx=0 tx=0).stop 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.533+0000 7f3ef039d740 1 -- 172.21.15.12:0/2787269523 shutdown_connections 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.533+0000 7f3ef039d740 1 --2- 172.21.15.12:0/2787269523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9c98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.533+0000 7f3ef039d740 1 --2- 172.21.15.12:0/2787269523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9e0970 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.533+0000 7f3ef039d740 1 --2- 172.21.15.12:0/2787269523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9cf340 0x55f5ee9cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.533+0000 7f3ef039d740 1 -- 172.21.15.12:0/2787269523 >> 172.21.15.12:0/2787269523 conn(0x55f5ee9ca8b0 msgr2=0x55f5ee9c91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.533+0000 7f3ef039d740 1 -- 172.21.15.12:0/2787269523 shutdown_connections 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.533+0000 7f3ef039d740 1 -- 172.21.15.12:0/2787269523 wait complete. 2024-03-17T23:55:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.534+0000 7f3ef039d740 1 -- start start 2024-03-17T23:55:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.534+0000 7f3ef039d740 1 Processor -- start 2024-03-17T23:55:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.534+0000 7f3ef039d740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cf340 0x55f5ee9dc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.534+0000 7f3ef039d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9dc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.534+0000 7f3ef039d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9df970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.534+0000 7f3ef039d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f5ee9e7000 con 0x55f5ee9d4ea0 2024-03-17T23:55:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.534+0000 7f3ef039d740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f5ee9e6930 con 0x55f5ee9cf340 2024-03-17T23:55:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.534+0000 7f3ef039d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f5ee9e76d0 con 0x55f5ee9cfd30 2024-03-17T23:55:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.534+0000 7f3ef039d740 10 client.0 _ll_register_callbacks cb 0x7f3ed00d4a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:55:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.534+0000 7f3ee6a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cf340 0x55f5ee9dc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3ee6a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cf340 0x55f5ee9dc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60670/0 (socket says 172.21.15.12:60670) 2024-03-17T23:55:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3ee6a00640 1 -- 172.21.15.12:0/1463669722 learned_addr learned my addr 172.21.15.12:0/1463669722 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3ee7400640 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9df970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3edfe00640 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9dc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3ee6a00640 1 -- 172.21.15.12:0/1463669722 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cfd30 msgr2=0x55f5ee9dc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:55:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3ee6a00640 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9dc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3ee6a00640 1 -- 172.21.15.12:0/1463669722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9d4ea0 msgr2=0x55f5ee9df970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:55:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3ee6a00640 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9df970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3ee6a00640 1 -- 172.21.15.12:0/1463669722 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5ee9dff60 con 0x55f5ee9cf340 2024-03-17T23:55:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3edfe00640 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9dc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:55:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3ee6a00640 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cf340 0x55f5ee9dc410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8009c00 tx=0x7f3ed8010930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=627403605344c6fb server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3ecbe00640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-03-17T23:55:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3ecbe00640 1 -- 172.21.15.12:0/1463669722 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80200f0 con 0x55f5ee9cf340 2024-03-17T23:55:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3ecbe00640 1 -- 172.21.15.12:0/1463669722 <== mon.2 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f3ed8020290 con 0x55f5ee9cf340 2024-03-17T23:55:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.535+0000 7f3ef039d740 1 -- 172.21.15.12:0/1463669722 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ed009b670 con 0x55f5ee9cf340 2024-03-17T23:55:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.536+0000 7f3ecbe00640 1 -- 172.21.15.12:0/1463669722 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f3ed80204d0 con 0x55f5ee9cf340 2024-03-17T23:55:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.536+0000 7f3ef039d740 10 client.24701 fetch_fsmap learned FSMap version 17 2024-03-17T23:55:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.536+0000 7f3ef039d740 1 -- 172.21.15.12:0/1463669722 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55f5eea548c0 con 0x55f5ee9cf340 2024-03-17T23:55:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.536+0000 7f3ecbe00640 1 -- 172.21.15.12:0/1463669722 <== mon.2 v2:172.21.15.79:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ed009b670 con 0x55f5ee9cf340 2024-03-17T23:55:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.536+0000 7f3ecbe00640 1 -- 172.21.15.12:0/1463669722 <== mon.2 v2:172.21.15.79:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f3ed801e3f0 con 0x55f5ee9cf340 2024-03-17T23:55:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.536+0000 7f3ef039d740 10 client.24701 fetch_fsmap finished waiting for FSMap version 17 2024-03-17T23:55:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.536+0000 7f3ef039d740 10 client.24701 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-17T23:55:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.536+0000 7f3ef039d740 20 client.24701 populate_metadata read hostname 'smithi012' 2024-03-17T23:55:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.536+0000 7f3ef039d740 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55f5eea594a0 0x55f5eea581b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.536+0000 7f3ef039d740 4 client.24701 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:55:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.536+0000 7f3ef039d740 1 -- 172.21.15.12:0/1463669722 --> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] -- command(tid 0: {"prefix": "status"}) -- 0x55f5ee941190 con 0x55f5eea594a0 2024-03-17T23:55:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.536+0000 7f3edfe00640 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55f5eea594a0 0x55f5eea581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3edfe00640 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55f5eea594a0 0x55f5eea581b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82e09d6c457e1702 server_cookie=55c811e42cfe7824 in_seq=0 out_seq=0 2024-03-17T23:55:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ecbe00640 10 client.24701 ms_handle_connect on v2:172.21.15.79:6838/2478742711 2024-03-17T23:55:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ecbe00640 1 -- 172.21.15.12:0/1463669722 <== mds.0 v2:172.21.15.79:6838/2478742711 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55f5ee941190 con 0x55f5eea594a0 2024-03-17T23:55:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ecbe00640 10 client.24701 handle_command_reply: tid=0 2024-03-17T23:55:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ef039d740 1 client.24701 shutdown 2024-03-17T23:55:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ef039d740 1 -- 172.21.15.12:0/1463669722 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cf340 msgr2=0x55f5ee9dc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ef039d740 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cf340 0x55f5ee9dc410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8009c00 tx=0x7f3ed8010930 comp rx=0 tx=0).stop 2024-03-17T23:55:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ef039d740 1 -- 172.21.15.12:0/1463669722 shutdown_connections 2024-03-17T23:55:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ef039d740 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55f5eea594a0 0x55f5eea581b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ef039d740 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f5ee9d4ea0 0x55f5ee9df970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ef039d740 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f5ee9cfd30 0x55f5ee9dc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ecbe00640 0 client.24701 ms_handle_reset on v2:172.21.15.79:6838/2478742711 2024-03-17T23:55:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ef039d740 1 --2- 172.21.15.12:0/1463669722 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f5ee9cf340 0x55f5ee9dc410 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ef039d740 1 -- 172.21.15.12:0/1463669722 >> 172.21.15.12:0/1463669722 conn(0x55f5ee9ca8b0 msgr2=0x55f5ee9c9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ef039d740 1 -- 172.21.15.12:0/1463669722 shutdown_connections 2024-03-17T23:55:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.537+0000 7f3ef039d740 1 -- 172.21.15.12:0/1463669722 wait complete. 2024-03-17T23:55:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.538+0000 7f3ef039d740 20 client.24701 trim_cache size 0 max 16384 2024-03-17T23:55:05.544 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-17T23:55:05.544 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:55:05.544 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-17T23:55:05.544 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24587, 2024-03-17T23:55:05.544 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-17T23:55:05.544 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-17T23:55:05.544 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-17T23:55:05.544 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 8.2277931150000008, 2024-03-17T23:55:05.544 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 17, 2024-03-17T23:55:05.544 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 90, 2024-03-17T23:55:05.544 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 90, 2024-03-17T23:55:05.544 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 15.244425452 2024-03-17T23:55:05.544 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-17T23:55:05.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.538+0000 7f3ee7e00640 1 -- 172.21.15.12:0/2688241340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ecc0a2880 msgr2=0x7f3ecc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.538+0000 7f3ee7e00640 1 --2- 172.21.15.12:0/2688241340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ecc0a2880 0x7f3ecc0a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0006010 tx=0x7f3ee00333e0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.538+0000 7f3ee7e00640 1 -- 172.21.15.12:0/2688241340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ee81017b0 msgr2=0x7f3ee8102c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:05.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.538+0000 7f3ee7e00640 1 --2- 172.21.15.12:0/2688241340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ee81017b0 0x7f3ee8102c00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000d950 tx=0x7f3ed000dd50 comp rx=0 tx=0).stop 2024-03-17T23:55:05.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.538+0000 7f3ee7e00640 1 -- 172.21.15.12:0/2688241340 shutdown_connections 2024-03-17T23:55:05.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.538+0000 7f3ee7e00640 1 --2- 172.21.15.12:0/2688241340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ecc0a2880 0x7f3ecc0a4d40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.538+0000 7f3ee7e00640 1 --2- 172.21.15.12:0/2688241340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee8102170 0x7f3ee8103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.538+0000 7f3ee7e00640 1 --2- 172.21.15.12:0/2688241340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ee81017b0 0x7f3ee8102c00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.538+0000 7f3ee7e00640 1 --2- 172.21.15.12:0/2688241340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ee81005c0 0x7f3ee81045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.538+0000 7f3ee7e00640 1 -- 172.21.15.12:0/2688241340 >> 172.21.15.12:0/2688241340 conn(0x7f3ee80fbb50 msgr2=0x7f3ee8068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.538+0000 7f3ee7e00640 1 -- 172.21.15.12:0/2688241340 shutdown_connections 2024-03-17T23:55:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:05.539+0000 7f3ee7e00640 1 -- 172.21.15.12:0/2688241340 wait complete. 2024-03-17T23:55:05.558 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-17T23:55:05.558 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:55:05.558 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-17T23:55:06.018 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:06.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:05 smithi012 ceph-mon[31541]: pgmap v325: 129 pgs: 2 creating+peering, 15 unknown, 112 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 719 B/s wr, 1 op/s 2024-03-17T23:55:06.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:05 smithi012 ceph-mon[31541]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-17T23:55:06.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:05 smithi012 ceph-mon[31541]: Cluster is now healthy 2024-03-17T23:55:06.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:05 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3674217268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:06.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:05 smithi079 ceph-mon[36241]: pgmap v325: 129 pgs: 2 creating+peering, 15 unknown, 112 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 719 B/s wr, 1 op/s 2024-03-17T23:55:06.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:05 smithi079 ceph-mon[36241]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-17T23:55:06.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:05 smithi079 ceph-mon[36241]: Cluster is now healthy 2024-03-17T23:55:06.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:05 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3674217268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:06.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:05 smithi191 ceph-mon[36708]: pgmap v325: 129 pgs: 2 creating+peering, 15 unknown, 112 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 719 B/s wr, 1 op/s 2024-03-17T23:55:06.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:05 smithi191 ceph-mon[36708]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-17T23:55:06.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:05 smithi191 ceph-mon[36708]: Cluster is now healthy 2024-03-17T23:55:06.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:05 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3674217268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:08.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:07 smithi012 ceph-mon[31541]: pgmap v326: 129 pgs: 2 creating+peering, 127 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-03-17T23:55:08.143 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:07 smithi191 ceph-mon[36708]: pgmap v326: 129 pgs: 2 creating+peering, 127 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-03-17T23:55:08.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:07 smithi079 ceph-mon[36241]: pgmap v326: 129 pgs: 2 creating+peering, 127 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-03-17T23:55:08.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.297+0000 7fde71200700 1 -- 172.21.15.12:0/3235920539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde6c1092a0 msgr2=0x7fde6c1157b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:08.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.297+0000 7fde71200700 1 --2- 172.21.15.12:0/3235920539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde6c1092a0 0x7fde6c1157b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fde6400b0a0 tx=0x7fde6400b3b0 comp rx=0 tx=0).stop 2024-03-17T23:55:08.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.297+0000 7fde71200700 1 -- 172.21.15.12:0/3235920539 shutdown_connections 2024-03-17T23:55:08.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.297+0000 7fde71200700 1 --2- 172.21.15.12:0/3235920539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde6c1092a0 0x7fde6c1157b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:08.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.297+0000 7fde71200700 1 --2- 172.21.15.12:0/3235920539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde6c10e8e0 0x7fde6c10ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:08.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.297+0000 7fde71200700 1 --2- 172.21.15.12:0/3235920539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde6c1088e0 0x7fde6c108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:08.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.297+0000 7fde71200700 1 -- 172.21.15.12:0/3235920539 >> 172.21.15.12:0/3235920539 conn(0x7fde6c075aa0 msgr2=0x7fde6c077ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:08.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.297+0000 7fde71200700 1 -- 172.21.15.12:0/3235920539 shutdown_connections 2024-03-17T23:55:08.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.297+0000 7fde71200700 1 -- 172.21.15.12:0/3235920539 wait complete. 2024-03-17T23:55:08.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.298+0000 7fde71200700 1 Processor -- start 2024-03-17T23:55:08.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.298+0000 7fde71200700 1 -- start start 2024-03-17T23:55:08.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde71200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde6c1088e0 0x7fde6c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:08.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde71200700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde6c1092a0 0x7fde6c19f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:08.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde71200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde6c10e8e0 0x7fde6c1a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:08.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde71200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c118cc0 con 0x7fde6c10e8e0 2024-03-17T23:55:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde71200700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c118b40 con 0x7fde6c1092a0 2024-03-17T23:55:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde71200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c118e40 con 0x7fde6c1088e0 2024-03-17T23:55:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde63e00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde6c1092a0 0x7fde6c19f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde6a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde6c10e8e0 0x7fde6c1a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde6a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde6c10e8e0 0x7fde6c1a38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42050/0 (socket says 172.21.15.12:42050) 2024-03-17T23:55:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde6a800700 1 -- 172.21.15.12:0/910056340 learned_addr learned my addr 172.21.15.12:0/910056340 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde63e00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde6c1092a0 0x7fde6c19f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60698/0 (socket says 172.21.15.12:60698) 2024-03-17T23:55:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde69e00700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde6c1088e0 0x7fde6c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde63e00700 1 -- 172.21.15.12:0/910056340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde6c1088e0 msgr2=0x7fde6c19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde63e00700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde6c1088e0 0x7fde6c19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde63e00700 1 -- 172.21.15.12:0/910056340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde6c10e8e0 msgr2=0x7fde6c1a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.299+0000 7fde63e00700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde6c10e8e0 0x7fde6c1a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.300+0000 7fde63e00700 1 -- 172.21.15.12:0/910056340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6c1a4010 con 0x7fde6c1092a0 2024-03-17T23:55:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.300+0000 7fde69e00700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde6c1088e0 0x7fde6c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:55:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.300+0000 7fde6a800700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde6c10e8e0 0x7fde6c1a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:55:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.300+0000 7fde63e00700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde6c1092a0 0x7fde6c19f590 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde5400e9f0 tx=0x7fde5400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.300+0000 7fde62a00700 1 -- 172.21.15.12:0/910056340 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5400cbd0 con 0x7fde6c1092a0 2024-03-17T23:55:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.300+0000 7fde62a00700 1 -- 172.21.15.12:0/910056340 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde5400cd70 con 0x7fde6c1092a0 2024-03-17T23:55:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.300+0000 7fde62a00700 1 -- 172.21.15.12:0/910056340 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54018710 con 0x7fde6c1092a0 2024-03-17T23:55:08.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.300+0000 7fde71200700 1 -- 172.21.15.12:0/910056340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde6c1a4300 con 0x7fde6c1092a0 2024-03-17T23:55:08.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.300+0000 7fde71200700 1 -- 172.21.15.12:0/910056340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde6c1abb80 con 0x7fde6c1092a0 2024-03-17T23:55:08.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.301+0000 7fde71200700 1 -- 172.21.15.12:0/910056340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde6c068f90 con 0x7fde6c1092a0 2024-03-17T23:55:08.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.303+0000 7fde62a00700 1 -- 172.21.15.12:0/910056340 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fde54010430 con 0x7fde6c1092a0 2024-03-17T23:55:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.304+0000 7fde62a00700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fde500a2910 0x7fde500a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.304+0000 7fde62a00700 1 -- 172.21.15.12:0/910056340 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fde540ceef0 con 0x7fde6c1092a0 2024-03-17T23:55:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.305+0000 7fde69e00700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fde500a2910 0x7fde500a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.305+0000 7fde69e00700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fde500a2910 0x7fde500a4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde5c005e10 tx=0x7fde5c005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:08.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.308+0000 7fde62a00700 1 -- 172.21.15.12:0/910056340 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fde5409bcf0 con 0x7fde6c1092a0 2024-03-17T23:55:08.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.589+0000 7fde71200700 1 -- 172.21.15.12:0/910056340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fde6c02db70 con 0x7fde6c1092a0 2024-03-17T23:55:08.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.589+0000 7fde62a00700 1 -- 172.21.15.12:0/910056340 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fde5409b5f0 con 0x7fde6c1092a0 2024-03-17T23:55:08.590 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-17T23:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.592+0000 7fde71200700 1 -- 172.21.15.12:0/910056340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fde500a2910 msgr2=0x7fde500a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.592+0000 7fde71200700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fde500a2910 0x7fde500a4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde5c005e10 tx=0x7fde5c005d60 comp rx=0 tx=0).stop 2024-03-17T23:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.592+0000 7fde71200700 1 -- 172.21.15.12:0/910056340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde6c1092a0 msgr2=0x7fde6c19f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.592+0000 7fde71200700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde6c1092a0 0x7fde6c19f590 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde5400e9f0 tx=0x7fde5400ed00 comp rx=0 tx=0).stop 2024-03-17T23:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.592+0000 7fde71200700 1 -- 172.21.15.12:0/910056340 shutdown_connections 2024-03-17T23:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.592+0000 7fde71200700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fde500a2910 0x7fde500a4dd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.592+0000 7fde71200700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde6c10e8e0 0x7fde6c1a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.592+0000 7fde71200700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde6c1088e0 0x7fde6c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.592+0000 7fde71200700 1 --2- 172.21.15.12:0/910056340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde6c1092a0 0x7fde6c19f590 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.592+0000 7fde71200700 1 -- 172.21.15.12:0/910056340 >> 172.21.15.12:0/910056340 conn(0x7fde6c075aa0 msgr2=0x7fde6c075fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.593+0000 7fde71200700 1 -- 172.21.15.12:0/910056340 shutdown_connections 2024-03-17T23:55:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:08.593+0000 7fde71200700 1 -- 172.21.15.12:0/910056340 wait complete. 2024-03-17T23:55:09.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:08 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/910056340' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-17T23:55:09.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:08 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/910056340' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-17T23:55:09.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:08 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/910056340' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-17T23:55:09.526 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-17T23:55:09.526 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-17T23:55:09.526 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-17T23:55:09.526 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-17T23:55:09.526 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:55:09.526 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-17T23:55:09.526 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-17T23:55:09.527 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-17T23:55:09.527 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-17T23:55:09.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:55:09.527 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-17T23:55:09.588 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:55:09.588 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:55:09.588 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:55:09.588 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:55:09.588 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-17T23:55:09.588 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:55:09.588 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:55:09.588 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:55:09.588 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-17T23:55:09.588 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:55:09.589 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:55:09.589 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-17T23:55:09.589 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-17T23:55:09.589 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-17T23:55:09.589 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-17T23:55:09.589 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-17T23:55:09.589 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 41622sec preferred_lft 41622sec 2024-03-17T23:55:09.589 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-03-17T23:55:09.589 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:55:09.589 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:55:09.589 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-17T23:55:09.589 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-17T23:55:09.590 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-17T23:55:09.591 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:55:09.591 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-17T23:55:09.591 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add name ceph-brx type bridge 2024-03-17T23:55:09.591 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr flush dev ceph-brx 2024-03-17T23:55:09.591 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set ceph-brx up 2024-03-17T23:55:09.591 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-17T23:55:09.591 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-17T23:55:09.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:55:10.028 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:09 smithi012 ceph-mon[31541]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 423 B/s wr, 0 op/s 2024-03-17T23:55:10.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:55:10.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:55:10.146 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-17T23:55:10.231 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-17T23:55:10.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:55:10.235 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-17T23:55:10.294 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-17T23:55:10.295 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-17T23:55:10.295 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-17T23:55:10.295 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-17T23:55:10.295 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-17T23:55:10.295 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:55:10.295 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-17T23:55:10.296 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-17T23:55:10.296 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-17T23:55:10.296 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-17T23:55:10.296 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-17T23:55:10.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:09 smithi079 ceph-mon[36241]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 423 B/s wr, 0 op/s 2024-03-17T23:55:10.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:09 smithi191 ceph-mon[36708]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 423 B/s wr, 0 op/s 2024-03-17T23:55:10.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:55:11.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:55:11.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:55:11.086 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-17T23:55:11.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:55:11.142 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list-id 2024-03-17T23:55:11.198 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:55:11.198 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-17T23:55:11.198 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:55:11.198 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-17T23:55:11.198 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-17T23:55:11.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:55:11.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:55:11.413 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-17T23:55:11.413 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:55:11.413 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-17T23:55:11.414 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-17T23:55:11.414 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-17T23:55:11.414 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-17T23:55:11.414 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-17T23:55:11.414 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-17T23:55:11.414 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-17T23:55:11.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:55:11.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:55:11.792 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:55:11.792 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-17T23:55:11.792 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-17T23:55:11.792 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set dev brx.0 master ceph-brx 2024-03-17T23:55:11.792 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-17T23:55:11.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:55:12.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:55:12.040 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-17T23:55:12.040 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:55:12.040 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:55:12.088 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-17T23:55:12.088 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:55:12.088 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:55:12.133 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-03-17T23:55:12.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:11 smithi012 ceph-mon[31541]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:55:12.278 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:55:12.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:11 smithi079 ceph-mon[36241]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:55:12.332 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-03-17T23:55:12.333 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-03-17T23:55:12.333 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-03-17T23:55:12.333 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-03-17T23:55:12.333 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-03-17T23:55:12.333 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-03-17T23:55:12.333 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-03-17T23:55:12.333 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-03-17T23:55:12.333 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-03-17T23:55:12.333 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-03-17T23:55:12.333 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-03-17T23:55:12.333 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-03-17T23:55:12.334 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:55:12.334 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-03-17T23:55:12.334 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-03-17T23:55:12.334 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-03-17T23:55:12.334 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-03-17T23:55:12.334 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-03-17T23:55:12.334 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:55:12.334 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:55:12.334 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-03-17T23:55:12.334 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:55:12.334 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:55:12.334 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-03-17T23:55:12.335 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-03-17T23:55:12.335 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay 2024-03-17T23:55:12.335 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/74869470276cf9f613a8b68f88583e423e5d4f3e72ed899e7b660008de59b034/merged 2024-03-17T23:55:12.335 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/161d100b2a6be8388b7ee1bda707236d47574a2060dccedc4c684f11eeeac232/merged 2024-03-17T23:55:12.335 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/0 2024-03-17T23:55:12.335 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:55:12.335 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/e8caa1b21d28f759836e995e435ff2e6ed857bc3b832edd6423b4f620466593f/merged 2024-03-17T23:55:12.335 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/dda7abade2a017cf6634041c28c011246e22d3d8c61f31a03fd2e8c7d228acbe/merged 2024-03-17T23:55:12.335 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/d477d410cbba929f716a4efbfc68ff6a8c6322671d39e6d4d24e3adfd854490d/merged 2024-03-17T23:55:12.335 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/56927d85c0dfd892fa3c12fd60bb29f4f569c81429341d19b66e52aaa20dc660/merged 2024-03-17T23:55:12.335 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/b67d3983bbc937c27949c843877ef4f413fbcce71a613dad4b407b9ae1d75999/merged 2024-03-17T23:55:12.335 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/605a62d934b03053f489141bb4a9f23abbd82524a086b79784c58facb19e155e/merged 2024-03-17T23:55:12.336 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/418e9903b0b7b63e360a7f5a7854f12379e244eb48a2721956cb318083adaac1/merged 2024-03-17T23:55:12.336 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/3ca26d89aa511235dff191e9d64ebeb23252104d779ddfc6a9cb843320f28c97/merged 2024-03-17T23:55:12.336 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-03-17T23:55:12.336 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:55:12.336 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:55:12.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:55:12.337 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-03-17T23:55:12.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:11 smithi191 ceph-mon[36708]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:55:12.394 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-17T23:55:12.394 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-17T23:55:12.399 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-03-17T23:55:12.475 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:55:12.531 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-03-17T23:55:12.531 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-03-17T23:55:12.531 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-03-17T23:55:12.531 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-03-17T23:55:12.531 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-03-17T23:55:12.531 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-03-17T23:55:12.531 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-03-17T23:55:12.531 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-03-17T23:55:12.531 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-03-17T23:55:12.531 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-03-17T23:55:12.531 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-03-17T23:55:12.532 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/74869470276cf9f613a8b68f88583e423e5d4f3e72ed899e7b660008de59b034/merged 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/161d100b2a6be8388b7ee1bda707236d47574a2060dccedc4c684f11eeeac232/merged 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/0 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/e8caa1b21d28f759836e995e435ff2e6ed857bc3b832edd6423b4f620466593f/merged 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/dda7abade2a017cf6634041c28c011246e22d3d8c61f31a03fd2e8c7d228acbe/merged 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/d477d410cbba929f716a4efbfc68ff6a8c6322671d39e6d4d24e3adfd854490d/merged 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/56927d85c0dfd892fa3c12fd60bb29f4f569c81429341d19b66e52aaa20dc660/merged 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/b67d3983bbc937c27949c843877ef4f413fbcce71a613dad4b407b9ae1d75999/merged 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/605a62d934b03053f489141bb4a9f23abbd82524a086b79784c58facb19e155e/merged 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/418e9903b0b7b63e360a7f5a7854f12379e244eb48a2721956cb318083adaac1/merged 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/3ca26d89aa511235dff191e9d64ebeb23252104d779ddfc6a9cb843320f28c97/merged 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-03-17T23:55:12.533 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:55:12.534 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:55:12.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:55:12.535 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-03-17T23:55:12.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi012.stderr:2024-03-17T23:55:12.598+0000 7f135ca41480 -1 init, newargv = 0x5601efb746b0 newargc=15 2024-03-17T23:55:12.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi012.stderr:ceph-fuse[72602]: starting ceph client 2024-03-17T23:55:12.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi012.stderr:ceph-fuse[72602]: starting fuse 2024-03-17T23:55:13.681 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-03-17T23:55:13.793 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:55:13.854 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-03-17T23:55:13.854 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-03-17T23:55:13.855 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/74869470276cf9f613a8b68f88583e423e5d4f3e72ed899e7b660008de59b034/merged 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/161d100b2a6be8388b7ee1bda707236d47574a2060dccedc4c684f11eeeac232/merged 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/0 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/e8caa1b21d28f759836e995e435ff2e6ed857bc3b832edd6423b4f620466593f/merged 2024-03-17T23:55:13.856 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/dda7abade2a017cf6634041c28c011246e22d3d8c61f31a03fd2e8c7d228acbe/merged 2024-03-17T23:55:13.857 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/d477d410cbba929f716a4efbfc68ff6a8c6322671d39e6d4d24e3adfd854490d/merged 2024-03-17T23:55:13.857 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/56927d85c0dfd892fa3c12fd60bb29f4f569c81429341d19b66e52aaa20dc660/merged 2024-03-17T23:55:13.857 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/b67d3983bbc937c27949c843877ef4f413fbcce71a613dad4b407b9ae1d75999/merged 2024-03-17T23:55:13.857 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/605a62d934b03053f489141bb4a9f23abbd82524a086b79784c58facb19e155e/merged 2024-03-17T23:55:13.857 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/418e9903b0b7b63e360a7f5a7854f12379e244eb48a2721956cb318083adaac1/merged 2024-03-17T23:55:13.857 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/3ca26d89aa511235dff191e9d64ebeb23252104d779ddfc6a9cb843320f28c97/merged 2024-03-17T23:55:13.857 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-03-17T23:55:13.857 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:55:13.857 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:55:13.857 INFO:teuthology.orchestra.run.smithi012.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-17T23:55:13.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:55:13.858 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-03-17T23:55:13.914 INFO:teuthology.orchestra.run.smithi012.stdout:118 2024-03-17T23:55:13.914 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [118] 2024-03-17T23:55:13.918 DEBUG:teuthology.orchestra.run.smithi012:> sudo stdin-killer -- python3 -c ' 2024-03-17T23:55:13.918 DEBUG:teuthology.orchestra.run.smithi012:> import glob 2024-03-17T23:55:13.918 DEBUG:teuthology.orchestra.run.smithi012:> import re 2024-03-17T23:55:13.918 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-03-17T23:55:13.918 DEBUG:teuthology.orchestra.run.smithi012:> import subprocess 2024-03-17T23:55:13.918 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-17T23:55:13.918 DEBUG:teuthology.orchestra.run.smithi012:> def _find_admin_socket(client_name): 2024-03-17T23:55:13.918 DEBUG:teuthology.orchestra.run.smithi012:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-17T23:55:13.918 DEBUG:teuthology.orchestra.run.smithi012:> files = glob.glob(asok_path) 2024-03-17T23:55:13.918 DEBUG:teuthology.orchestra.run.smithi012:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-17T23:55:13.918 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> # Given a non-glob path, it better be there 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> if "*" not in asok_path: 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> assert(len(files) == 1) 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> return files[0] 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> for f in files: 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> contents = proc_f.read() 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> if mountpoint in contents: 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> return f 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> print(_find_admin_socket("client.admin")) 2024-03-17T23:55:13.919 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-03-17T23:55:14.109 INFO:teuthology.orchestra.run.smithi012.stdout:/var/run/ceph/ceph-client.admin.72602.asok 2024-03-17T23:55:14.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:55:14.124 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:13 smithi012 ceph-mon[31541]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:55:14.128 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.72602.asok 2024-03-17T23:55:14.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:55:14.129 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.72602.asok status 2024-03-17T23:55:14.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:13 smithi079 ceph-mon[36241]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:55:14.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:13 smithi191 ceph-mon[36708]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:55:14.371 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-17T23:55:14.371 INFO:teuthology.orchestra.run.smithi012.stdout: "metadata": { 2024-03-17T23:55:14.371 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-17T23:55:14.371 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "entity_id": "admin", 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": "smithi012", 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "pid": "72602", 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "root": "/" 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_count": 0, 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_pinned_count": 0, 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24713, 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "inst": { 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "name": { 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "client", 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 24713 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-03-17T23:55:14.372 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3695638857 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3695638857 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: "inst_str": "client.24713 172.21.15.12:0/3695638857", 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: "addr_str": "172.21.15.12:0/3695638857", 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: "inode_count": 1, 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_epoch": 17, 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch": 90, 2024-03-17T23:55:14.373 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch_barrier": 0, 2024-03-17T23:55:14.374 INFO:teuthology.orchestra.run.smithi012.stdout: "blocklisted": false, 2024-03-17T23:55:14.374 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs" 2024-03-17T23:55:14.374 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-17T23:55:14.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:55:14.386 DEBUG:teuthology.orchestra.run.smithi012:> stat --file-system '--printf=%T 2024-03-17T23:55:14.386 DEBUG:teuthology.orchestra.run.smithi012:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:55:14.441 INFO:teuthology.orchestra.run.smithi012.stdout:fuseblk 2024-03-17T23:55:14.441 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:55:14.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:55:14.441 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:55:14.543 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-17T23:55:14.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-17T23:55:14.696 INFO:teuthology.orchestra.run.smithi012.stdout:# file: . 2024-03-17T23:55:14.696 INFO:teuthology.orchestra.run.smithi012.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-17T23:55:14.696 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:55:14.706 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:55:14.707 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:55:14.761 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:55:14.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.986+0000 7fc626000640 1 -- 172.21.15.12:0/33071395 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6200ffe60 msgr2=0x7fc620102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:14.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.986+0000 7fc626000640 1 --2- 172.21.15.12:0/33071395 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6200ffe60 0x7fc620102290 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc610009ca0 tx=0x7fc61002c100 comp rx=0 tx=0).stop 2024-03-17T23:55:14.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.986+0000 7fc626000640 1 -- 172.21.15.12:0/33071395 shutdown_connections 2024-03-17T23:55:14.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.986+0000 7fc626000640 1 --2- 172.21.15.12:0/33071395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc620105180 0x7fc620105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:14.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.986+0000 7fc626000640 1 --2- 172.21.15.12:0/33071395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6201027d0 0x7fc620104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:14.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.986+0000 7fc626000640 1 --2- 172.21.15.12:0/33071395 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6200ffe60 0x7fc620102290 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:14.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.986+0000 7fc626000640 1 -- 172.21.15.12:0/33071395 >> 172.21.15.12:0/33071395 conn(0x7fc6200fba60 msgr2=0x7fc6200fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:14.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.986+0000 7fc626000640 1 -- 172.21.15.12:0/33071395 shutdown_connections 2024-03-17T23:55:14.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.986+0000 7fc626000640 1 -- 172.21.15.12:0/33071395 wait complete. 2024-03-17T23:55:14.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.987+0000 7fc626000640 1 Processor -- start 2024-03-17T23:55:14.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.987+0000 7fc626000640 1 -- start start 2024-03-17T23:55:14.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.987+0000 7fc626000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6200ffe60 0x7fc62019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:14.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.987+0000 7fc626000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6201027d0 0x7fc62019e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:14.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.987+0000 7fc626000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc620105180 0x7fc6201a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:14.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.987+0000 7fc626000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc620113860 con 0x7fc620105180 2024-03-17T23:55:14.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.987+0000 7fc626000640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc6201136e0 con 0x7fc6200ffe60 2024-03-17T23:55:14.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.987+0000 7fc626000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc6201139e0 con 0x7fc6201027d0 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.988+0000 7fc624c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6200ffe60 0x7fc62019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.988+0000 7fc624c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6200ffe60 0x7fc62019de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:52868/0 (socket says 172.21.15.12:52868) 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.988+0000 7fc624c00640 1 -- 172.21.15.12:0/3356954103 learned_addr learned my addr 172.21.15.12:0/3356954103 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.988+0000 7fc625600640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc620105180 0x7fc6201a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.988+0000 7fc61fe00640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6201027d0 0x7fc62019e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.988+0000 7fc624c00640 1 -- 172.21.15.12:0/3356954103 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6201027d0 msgr2=0x7fc62019e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.988+0000 7fc624c00640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6201027d0 0x7fc62019e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.988+0000 7fc624c00640 1 -- 172.21.15.12:0/3356954103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc620105180 msgr2=0x7fc6201a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.988+0000 7fc624c00640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc620105180 0x7fc6201a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.988+0000 7fc624c00640 1 -- 172.21.15.12:0/3356954103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6201a5a80 con 0x7fc6200ffe60 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.988+0000 7fc625600640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc620105180 0x7fc6201a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.988+0000 7fc624c00640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6200ffe60 0x7fc62019de60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc6100325f0 tx=0x7fc610035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.989+0000 7fc61d600640 1 -- 172.21.15.12:0/3356954103 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc610039040 con 0x7fc6200ffe60 2024-03-17T23:55:14.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.989+0000 7fc61d600640 1 -- 172.21.15.12:0/3356954103 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc610044070 con 0x7fc6200ffe60 2024-03-17T23:55:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.989+0000 7fc61d600640 1 -- 172.21.15.12:0/3356954103 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc610004540 con 0x7fc6200ffe60 2024-03-17T23:55:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.989+0000 7fc626000640 1 -- 172.21.15.12:0/3356954103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6201a5d10 con 0x7fc6200ffe60 2024-03-17T23:55:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.989+0000 7fc626000640 1 -- 172.21.15.12:0/3356954103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6201a6180 con 0x7fc6200ffe60 2024-03-17T23:55:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.990+0000 7fc626000640 1 -- 172.21.15.12:0/3356954103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6200fd550 con 0x7fc6200ffe60 2024-03-17T23:55:14.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.991+0000 7fc61d600640 1 -- 172.21.15.12:0/3356954103 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc610032890 con 0x7fc6200ffe60 2024-03-17T23:55:14.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.993+0000 7fc61d600640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc6040a2800 0x7fc6040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:14.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.993+0000 7fc61d600640 1 -- 172.21.15.12:0/3356954103 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc6100f1460 con 0x7fc6200ffe60 2024-03-17T23:55:14.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.993+0000 7fc61fe00640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc6040a2800 0x7fc6040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:14.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.994+0000 7fc61fe00640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc6040a2800 0x7fc6040a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc62019f370 tx=0x7fc608005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:14.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:14.997+0000 7fc61d600640 1 -- 172.21.15.12:0/3356954103 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc6100be260 con 0x7fc6200ffe60 2024-03-17T23:55:15.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.204+0000 7fc626000640 1 -- 172.21.15.12:0/3356954103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6200623e0 con 0x7fc6200ffe60 2024-03-17T23:55:15.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.206+0000 7fc61d600640 1 -- 172.21.15.12:0/3356954103 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24044 (secure 0 0 0) 0x7fc6100bdb60 con 0x7fc6200ffe60 2024-03-17T23:55:15.207 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:15.207 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":90,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:55:02.337441+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:55:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.208+0000 7fc626000640 1 -- 172.21.15.12:0/3356954103 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc6040a2800 msgr2=0x7fc6040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.208+0000 7fc626000640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc6040a2800 0x7fc6040a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc62019f370 tx=0x7fc608005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.208+0000 7fc626000640 1 -- 172.21.15.12:0/3356954103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6200ffe60 msgr2=0x7fc62019de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.208+0000 7fc626000640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6200ffe60 0x7fc62019de60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc6100325f0 tx=0x7fc610035d10 comp rx=0 tx=0).stop 2024-03-17T23:55:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.208+0000 7fc626000640 1 -- 172.21.15.12:0/3356954103 shutdown_connections 2024-03-17T23:55:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.208+0000 7fc626000640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc6040a2800 0x7fc6040a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.208+0000 7fc626000640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc620105180 0x7fc6201a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.208+0000 7fc626000640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6201027d0 0x7fc62019e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.209+0000 7fc626000640 1 --2- 172.21.15.12:0/3356954103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6200ffe60 0x7fc62019de60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.209+0000 7fc626000640 1 -- 172.21.15.12:0/3356954103 >> 172.21.15.12:0/3356954103 conn(0x7fc6200fba60 msgr2=0x7fc620103160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.209+0000 7fc626000640 1 -- 172.21.15.12:0/3356954103 shutdown_connections 2024-03-17T23:55:15.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.209+0000 7fc626000640 1 -- 172.21.15.12:0/3356954103 wait complete. 2024-03-17T23:55:15.231 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:49:38.640282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:54:54.540345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:54:55.803926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:55:00.355066+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:55:15.232 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:55:15.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.458+0000 7f574fe00640 1 -- 172.21.15.12:0/1193113778 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5750103290 msgr2=0x7f5750109b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:15.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.458+0000 7f574fe00640 1 --2- 172.21.15.12:0/1193113778 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5750103290 0x7f5750109b20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5748009ca0 tx=0x7f574802c100 comp rx=0 tx=0).stop 2024-03-17T23:55:15.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.459+0000 7f574fe00640 1 -- 172.21.15.12:0/1193113778 shutdown_connections 2024-03-17T23:55:15.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.459+0000 7f574fe00640 1 --2- 172.21.15.12:0/1193113778 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5750103290 0x7f5750109b20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.459+0000 7f574fe00640 1 --2- 172.21.15.12:0/1193113778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57501028d0 0x7f5750102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.459+0000 7f574fe00640 1 --2- 172.21.15.12:0/1193113778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57501016e0 0x7f5750101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.459+0000 7f574fe00640 1 -- 172.21.15.12:0/1193113778 >> 172.21.15.12:0/1193113778 conn(0x7f57500fcc50 msgr2=0x7f57500ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:15.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.459+0000 7f574fe00640 1 -- 172.21.15.12:0/1193113778 shutdown_connections 2024-03-17T23:55:15.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.459+0000 7f574fe00640 1 -- 172.21.15.12:0/1193113778 wait complete. 2024-03-17T23:55:15.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.460+0000 7f574fe00640 1 Processor -- start 2024-03-17T23:55:15.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.460+0000 7f574fe00640 1 -- start start 2024-03-17T23:55:15.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57501016e0 0x7f5750199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:15.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574fe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57501028d0 0x7f575019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:15.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5750103290 0x7f57501a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:15.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574fe00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f575010d360 con 0x7f5750103290 2024-03-17T23:55:15.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574fe00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f575010d1e0 con 0x7f57501028d0 2024-03-17T23:55:15.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574fe00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f575010d4e0 con 0x7f57501016e0 2024-03-17T23:55:15.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574e000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57501028d0 0x7f575019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:15.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574e000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57501028d0 0x7f575019a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:52886/0 (socket says 172.21.15.12:52886) 2024-03-17T23:55:15.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574e000640 1 -- 172.21.15.12:0/2329955602 learned_addr learned my addr 172.21.15.12:0/2329955602 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:15.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574f400640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5750103290 0x7f57501a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:15.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574ea00640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57501016e0 0x7f5750199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:15.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574e000640 1 -- 172.21.15.12:0/2329955602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57501016e0 msgr2=0x7f5750199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:15.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574e000640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57501016e0 0x7f5750199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574e000640 1 -- 172.21.15.12:0/2329955602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5750103290 msgr2=0x7f57501a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:15.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574e000640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5750103290 0x7f57501a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.461+0000 7f574e000640 1 -- 172.21.15.12:0/2329955602 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57501a17b0 con 0x7f57501028d0 2024-03-17T23:55:15.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.462+0000 7f574e000640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57501028d0 0x7f575019a0d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f573c00e9f0 tx=0x7f573c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:15.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.462+0000 7f5747400640 1 -- 172.21.15.12:0/2329955602 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c00cd80 con 0x7f57501028d0 2024-03-17T23:55:15.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.462+0000 7f5747400640 1 -- 172.21.15.12:0/2329955602 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f573c005500 con 0x7f57501028d0 2024-03-17T23:55:15.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.462+0000 7f5747400640 1 -- 172.21.15.12:0/2329955602 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c010690 con 0x7f57501028d0 2024-03-17T23:55:15.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.462+0000 7f574fe00640 1 -- 172.21.15.12:0/2329955602 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57501a1aa0 con 0x7f57501028d0 2024-03-17T23:55:15.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.462+0000 7f574fe00640 1 -- 172.21.15.12:0/2329955602 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57501a1ff0 con 0x7f57501028d0 2024-03-17T23:55:15.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.463+0000 7f574fe00640 1 -- 172.21.15.12:0/2329955602 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57501a2280 con 0x7f57501028d0 2024-03-17T23:55:15.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.465+0000 7f5747400640 1 -- 172.21.15.12:0/2329955602 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f573c0046c0 con 0x7f57501028d0 2024-03-17T23:55:15.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.467+0000 7f5747400640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f57340a2880 0x7f57340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:15.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.467+0000 7f5747400640 1 -- 172.21.15.12:0/2329955602 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f573c0cf060 con 0x7f57501028d0 2024-03-17T23:55:15.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.467+0000 7f574ea00640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f57340a2880 0x7f57340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:15.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.468+0000 7f574ea00640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f57340a2880 0x7f57340a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5750102730 tx=0x7f5740005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:15.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.471+0000 7f5747400640 1 -- 172.21.15.12:0/2329955602 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f573c09be60 con 0x7f57501028d0 2024-03-17T23:55:15.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.673+0000 7f574fe00640 1 -- 172.21.15.12:0/2329955602 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f57501a2450 con 0x7f57501028d0 2024-03-17T23:55:15.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.674+0000 7f5747400640 1 -- 172.21.15.12:0/2329955602 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f573c09b760 con 0x7f57501028d0 2024-03-17T23:55:15.674 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-17T23:55:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.677+0000 7f574fe00640 1 -- 172.21.15.12:0/2329955602 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f57340a2880 msgr2=0x7f57340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.677+0000 7f574fe00640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f57340a2880 0x7f57340a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5750102730 tx=0x7f5740005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.677+0000 7f574fe00640 1 -- 172.21.15.12:0/2329955602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57501028d0 msgr2=0x7f575019a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.677+0000 7f574fe00640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57501028d0 0x7f575019a0d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f573c00e9f0 tx=0x7f573c00edf0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.677+0000 7f574fe00640 1 -- 172.21.15.12:0/2329955602 shutdown_connections 2024-03-17T23:55:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.677+0000 7f574fe00640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f57340a2880 0x7f57340a4d40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.677+0000 7f574fe00640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5750103290 0x7f57501a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.677+0000 7f574fe00640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57501028d0 0x7f575019a0d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.677+0000 7f574fe00640 1 --2- 172.21.15.12:0/2329955602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57501016e0 0x7f5750199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.677+0000 7f574fe00640 1 -- 172.21.15.12:0/2329955602 >> 172.21.15.12:0/2329955602 conn(0x7f57500fcc50 msgr2=0x7f5750073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.677+0000 7f574fe00640 1 -- 172.21.15.12:0/2329955602 shutdown_connections 2024-03-17T23:55:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.677+0000 7f574fe00640 1 -- 172.21.15.12:0/2329955602 wait complete. 2024-03-17T23:55:15.695 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:55:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.918+0000 7f1507200640 1 -- 172.21.15.12:0/3502747324 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1500104af0 msgr2=0x7f1500104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.918+0000 7f1507200640 1 --2- 172.21.15.12:0/3502747324 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1500104af0 0x7f1500104f70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f14f0009900 tx=0x7f14f002bc40 comp rx=0 tx=0).stop 2024-03-17T23:55:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.918+0000 7f1507200640 1 -- 172.21.15.12:0/3502747324 shutdown_connections 2024-03-17T23:55:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.918+0000 7f1507200640 1 --2- 172.21.15.12:0/3502747324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15001054b0 0x7f150010bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.918+0000 7f1507200640 1 --2- 172.21.15.12:0/3502747324 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1500104af0 0x7f1500104f70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.918+0000 7f1507200640 1 --2- 172.21.15.12:0/3502747324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1500103900 0x7f1500103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.918+0000 7f1507200640 1 -- 172.21.15.12:0/3502747324 >> 172.21.15.12:0/3502747324 conn(0x7f15000fee90 msgr2=0x7f15001012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.918+0000 7f1507200640 1 -- 172.21.15.12:0/3502747324 shutdown_connections 2024-03-17T23:55:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.918+0000 7f1507200640 1 -- 172.21.15.12:0/3502747324 wait complete. 2024-03-17T23:55:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.920+0000 7f1507200640 1 Processor -- start 2024-03-17T23:55:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.920+0000 7f1507200640 1 -- start start 2024-03-17T23:55:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.920+0000 7f1507200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1500103900 0x7f1500199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.920+0000 7f1507200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1500104af0 0x7f150019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.920+0000 7f1507200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15001054b0 0x7f15001a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.920+0000 7f1507200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f150010f5d0 con 0x7f1500103900 2024-03-17T23:55:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.920+0000 7f1507200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f150010f450 con 0x7f1500104af0 2024-03-17T23:55:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.920+0000 7f1507200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f150010f750 con 0x7f15001054b0 2024-03-17T23:55:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.921+0000 7f1505e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1500103900 0x7f1500199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.921+0000 7f1505400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1500104af0 0x7f150019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.921+0000 7f1505400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1500104af0 0x7f150019a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:52902/0 (socket says 172.21.15.12:52902) 2024-03-17T23:55:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.921+0000 7f1505400640 1 -- 172.21.15.12:0/4262176532 learned_addr learned my addr 172.21.15.12:0/4262176532 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.921+0000 7f1506800640 1 --2- 172.21.15.12:0/4262176532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15001054b0 0x7f15001a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.921+0000 7f1505400640 1 -- 172.21.15.12:0/4262176532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15001054b0 msgr2=0x7f15001a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.921+0000 7f1505400640 1 --2- 172.21.15.12:0/4262176532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15001054b0 0x7f15001a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.921+0000 7f1505400640 1 -- 172.21.15.12:0/4262176532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1500103900 msgr2=0x7f1500199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.921+0000 7f1505400640 1 --2- 172.21.15.12:0/4262176532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1500103900 0x7f1500199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.921+0000 7f1505400640 1 -- 172.21.15.12:0/4262176532 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15001a1860 con 0x7f1500104af0 2024-03-17T23:55:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.921+0000 7f1505400640 1 --2- 172.21.15.12:0/4262176532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1500104af0 0x7f150019a190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f14f0005ab0 tx=0x7f14f0005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.922+0000 7f14faa00640 1 -- 172.21.15.12:0/4262176532 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f0041070 con 0x7f1500104af0 2024-03-17T23:55:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.922+0000 7f14faa00640 1 -- 172.21.15.12:0/4262176532 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14f00054e0 con 0x7f1500104af0 2024-03-17T23:55:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.922+0000 7f1507200640 1 -- 172.21.15.12:0/4262176532 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15001a1af0 con 0x7f1500104af0 2024-03-17T23:55:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.922+0000 7f14faa00640 1 -- 172.21.15.12:0/4262176532 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f0034440 con 0x7f1500104af0 2024-03-17T23:55:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.922+0000 7f1507200640 1 -- 172.21.15.12:0/4262176532 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15001a2030 con 0x7f1500104af0 2024-03-17T23:55:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.923+0000 7f1507200640 1 -- 172.21.15.12:0/4262176532 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f150010abc0 con 0x7f1500104af0 2024-03-17T23:55:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.924+0000 7f14faa00640 1 -- 172.21.15.12:0/4262176532 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f14f0044070 con 0x7f1500104af0 2024-03-17T23:55:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.926+0000 7f14faa00640 1 --2- 172.21.15.12:0/4262176532 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14e80a2880 0x7f14e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.926+0000 7f14faa00640 1 -- 172.21.15.12:0/4262176532 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f14f00f0820 con 0x7f1500104af0 2024-03-17T23:55:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.926+0000 7f1505e00640 1 --2- 172.21.15.12:0/4262176532 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14e80a2880 0x7f14e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:15.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.927+0000 7f1505e00640 1 --2- 172.21.15.12:0/4262176532 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14e80a2880 0x7f14e80a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f150019b120 tx=0x7f14f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:15.930+0000 7f14faa00640 1 -- 172.21.15.12:0/4262176532 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f14f00bd030 con 0x7f1500104af0 2024-03-17T23:55:16.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.131+0000 7f1507200640 1 -- 172.21.15.12:0/4262176532 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f150002d1b0 con 0x7f1500104af0 2024-03-17T23:55:16.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.132+0000 7f14faa00640 1 -- 172.21.15.12:0/4262176532 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f14f00c10d0 con 0x7f1500104af0 2024-03-17T23:55:16.133 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-17T23:55:16.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.135+0000 7f1507200640 1 -- 172.21.15.12:0/4262176532 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14e80a2880 msgr2=0x7f14e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:16.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.135+0000 7f1507200640 1 --2- 172.21.15.12:0/4262176532 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14e80a2880 0x7f14e80a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f150019b120 tx=0x7f14f4005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:16.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.135+0000 7f1507200640 1 -- 172.21.15.12:0/4262176532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1500104af0 msgr2=0x7f150019a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:16.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.135+0000 7f1507200640 1 --2- 172.21.15.12:0/4262176532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1500104af0 0x7f150019a190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f14f0005ab0 tx=0x7f14f0005dd0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.135+0000 7f1507200640 1 -- 172.21.15.12:0/4262176532 shutdown_connections 2024-03-17T23:55:16.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.135+0000 7f1507200640 1 --2- 172.21.15.12:0/4262176532 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14e80a2880 0x7f14e80a4d40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.135+0000 7f1507200640 1 --2- 172.21.15.12:0/4262176532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15001054b0 0x7f15001a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.135+0000 7f1507200640 1 --2- 172.21.15.12:0/4262176532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1500104af0 0x7f150019a190 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.135+0000 7f1507200640 1 --2- 172.21.15.12:0/4262176532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1500103900 0x7f1500199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.135+0000 7f1507200640 1 -- 172.21.15.12:0/4262176532 >> 172.21.15.12:0/4262176532 conn(0x7f15000fee90 msgr2=0x7f1500107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:16.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.135+0000 7f1507200640 1 -- 172.21.15.12:0/4262176532 shutdown_connections 2024-03-17T23:55:16.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.135+0000 7f1507200640 1 -- 172.21.15.12:0/4262176532 wait complete. 2024-03-17T23:55:16.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:15 smithi012 ceph-mon[31541]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:55:16.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:15 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3356954103' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:16.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:15 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2329955602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:55:16.154 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:55:16.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:15 smithi079 ceph-mon[36241]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:55:16.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:15 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3356954103' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:16.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:15 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2329955602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:55:16.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:15 smithi191 ceph-mon[36708]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-17T23:55:16.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:15 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3356954103' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:16.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:15 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2329955602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:55:16.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.374+0000 7fe45ce00640 1 -- 172.21.15.12:0/3325756350 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe458107d00 msgr2=0x7fe458108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:16.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.374+0000 7fe45ce00640 1 --2- 172.21.15.12:0/3325756350 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe458107d00 0x7fe458108120 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe448009ca0 tx=0x7fe44802c100 comp rx=0 tx=0).stop 2024-03-17T23:55:16.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.374+0000 7fe45ce00640 1 -- 172.21.15.12:0/3325756350 shutdown_connections 2024-03-17T23:55:16.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.374+0000 7fe45ce00640 1 --2- 172.21.15.12:0/3325756350 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4581098b0 0x7fe458110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.374+0000 7fe45ce00640 1 --2- 172.21.15.12:0/3325756350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe458108ef0 0x7fe458109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.374+0000 7fe45ce00640 1 --2- 172.21.15.12:0/3325756350 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe458107d00 0x7fe458108120 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.374+0000 7fe45ce00640 1 -- 172.21.15.12:0/3325756350 >> 172.21.15.12:0/3325756350 conn(0x7fe458076cb0 msgr2=0x7fe4580790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:16.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.374+0000 7fe45ce00640 1 -- 172.21.15.12:0/3325756350 shutdown_connections 2024-03-17T23:55:16.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.374+0000 7fe45ce00640 1 -- 172.21.15.12:0/3325756350 wait complete. 2024-03-17T23:55:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.375+0000 7fe45ce00640 1 Processor -- start 2024-03-17T23:55:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.375+0000 7fe45ce00640 1 -- start start 2024-03-17T23:55:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.375+0000 7fe45ce00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe458107d00 0x7fe45819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.375+0000 7fe45ce00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe458108ef0 0x7fe45819e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.375+0000 7fe45ce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4581098b0 0x7fe4581a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.375+0000 7fe45ce00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe458113950 con 0x7fe4581098b0 2024-03-17T23:55:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.375+0000 7fe45ce00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe4581137d0 con 0x7fe458108ef0 2024-03-17T23:55:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.375+0000 7fe45ce00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe458113ad0 con 0x7fe458107d00 2024-03-17T23:55:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.376+0000 7fe44ea00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe458108ef0 0x7fe45819e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:16.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.376+0000 7fe457400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe458107d00 0x7fe45819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.376+0000 7fe44ea00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe458108ef0 0x7fe45819e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:52918/0 (socket says 172.21.15.12:52918) 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.376+0000 7fe44ea00640 1 -- 172.21.15.12:0/3929776702 learned_addr learned my addr 172.21.15.12:0/3929776702 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.376+0000 7fe457e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4581098b0 0x7fe4581a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.376+0000 7fe44ea00640 1 -- 172.21.15.12:0/3929776702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe458107d00 msgr2=0x7fe45819e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.376+0000 7fe44ea00640 1 --2- 172.21.15.12:0/3929776702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe458107d00 0x7fe45819e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.376+0000 7fe44ea00640 1 -- 172.21.15.12:0/3929776702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4581098b0 msgr2=0x7fe4581a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.376+0000 7fe44ea00640 1 --2- 172.21.15.12:0/3929776702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4581098b0 0x7fe4581a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.376+0000 7fe44ea00640 1 -- 172.21.15.12:0/3929776702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4581a5c20 con 0x7fe458108ef0 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.376+0000 7fe44ea00640 1 --2- 172.21.15.12:0/3929776702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe458108ef0 0x7fe45819e540 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe44000e9f0 tx=0x7fe44000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.377+0000 7fe454c00640 1 -- 172.21.15.12:0/3929776702 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44000cd80 con 0x7fe458108ef0 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.377+0000 7fe454c00640 1 -- 172.21.15.12:0/3929776702 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe440010910 con 0x7fe458108ef0 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.377+0000 7fe454c00640 1 -- 172.21.15.12:0/3929776702 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe440018990 con 0x7fe458108ef0 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.377+0000 7fe45ce00640 1 -- 172.21.15.12:0/3929776702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4581a5f10 con 0x7fe458108ef0 2024-03-17T23:55:16.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.377+0000 7fe45ce00640 1 -- 172.21.15.12:0/3929776702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4581a6380 con 0x7fe458108ef0 2024-03-17T23:55:16.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.378+0000 7fe45ce00640 1 -- 172.21.15.12:0/3929776702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe45810f0a0 con 0x7fe458108ef0 2024-03-17T23:55:16.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.380+0000 7fe454c00640 1 -- 172.21.15.12:0/3929776702 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe440010ac0 con 0x7fe458108ef0 2024-03-17T23:55:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.381+0000 7fe454c00640 1 --2- 172.21.15.12:0/3929776702 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe43c0a2880 0x7fe43c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.381+0000 7fe454c00640 1 -- 172.21.15.12:0/3929776702 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe4400cef80 con 0x7fe458108ef0 2024-03-17T23:55:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.381+0000 7fe457400640 1 --2- 172.21.15.12:0/3929776702 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe43c0a2880 0x7fe43c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:16.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.382+0000 7fe457400640 1 --2- 172.21.15.12:0/3929776702 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe43c0a2880 0x7fe43c0a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe44802c640 tx=0x7fe448009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:16.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.389+0000 7fe454c00640 1 -- 172.21.15.12:0/3929776702 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe44009b730 con 0x7fe458108ef0 2024-03-17T23:55:16.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.592+0000 7fe45ce00640 1 -- 172.21.15.12:0/3929776702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe4580623e0 con 0x7fe458108ef0 2024-03-17T23:55:16.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.593+0000 7fe454c00640 1 -- 172.21.15.12:0/3929776702 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fe44009f7d0 con 0x7fe458108ef0 2024-03-17T23:55:16.594 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-17T23:55:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.596+0000 7fe45ce00640 1 -- 172.21.15.12:0/3929776702 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe43c0a2880 msgr2=0x7fe43c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.596+0000 7fe45ce00640 1 --2- 172.21.15.12:0/3929776702 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe43c0a2880 0x7fe43c0a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe44802c640 tx=0x7fe448009360 comp rx=0 tx=0).stop 2024-03-17T23:55:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.596+0000 7fe45ce00640 1 -- 172.21.15.12:0/3929776702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe458108ef0 msgr2=0x7fe45819e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.596+0000 7fe45ce00640 1 --2- 172.21.15.12:0/3929776702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe458108ef0 0x7fe45819e540 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe44000e9f0 tx=0x7fe44000eeb0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.596+0000 7fe45ce00640 1 -- 172.21.15.12:0/3929776702 shutdown_connections 2024-03-17T23:55:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.596+0000 7fe45ce00640 1 --2- 172.21.15.12:0/3929776702 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe43c0a2880 0x7fe43c0a4d40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.596+0000 7fe45ce00640 1 --2- 172.21.15.12:0/3929776702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4581098b0 0x7fe4581a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.596+0000 7fe45ce00640 1 --2- 172.21.15.12:0/3929776702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe458108ef0 0x7fe45819e540 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.596+0000 7fe45ce00640 1 --2- 172.21.15.12:0/3929776702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe458107d00 0x7fe45819e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.596+0000 7fe45ce00640 1 -- 172.21.15.12:0/3929776702 >> 172.21.15.12:0/3929776702 conn(0x7fe458076cb0 msgr2=0x7fe45810c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.596+0000 7fe45ce00640 1 -- 172.21.15.12:0/3929776702 shutdown_connections 2024-03-17T23:55:16.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.596+0000 7fe45ce00640 1 -- 172.21.15.12:0/3929776702 wait complete. 2024-03-17T23:55:16.617 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:55:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.837+0000 7f0922c00640 1 -- 172.21.15.12:0/1929508055 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f091c0773e0 msgr2=0x7f091c077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.837+0000 7f0922c00640 1 --2- 172.21.15.12:0/1929508055 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f091c0773e0 0x7f091c077840 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f090c009860 tx=0x7f090c02bb70 comp rx=0 tx=0).stop 2024-03-17T23:55:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.837+0000 7f0922c00640 1 -- 172.21.15.12:0/1929508055 shutdown_connections 2024-03-17T23:55:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.837+0000 7f0922c00640 1 --2- 172.21.15.12:0/1929508055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c077d80 0x7f091c078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.837+0000 7f0922c00640 1 --2- 172.21.15.12:0/1929508055 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f091c0773e0 0x7f091c077840 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.837+0000 7f0922c00640 1 --2- 172.21.15.12:0/1929508055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f091c0761a0 0x7f091c0765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.837+0000 7f0922c00640 1 -- 172.21.15.12:0/1929508055 >> 172.21.15.12:0/1929508055 conn(0x7f091c0fee50 msgr2=0x7f091c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.837+0000 7f0922c00640 1 -- 172.21.15.12:0/1929508055 shutdown_connections 2024-03-17T23:55:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.837+0000 7f0922c00640 1 -- 172.21.15.12:0/1929508055 wait complete. 2024-03-17T23:55:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.838+0000 7f0922c00640 1 Processor -- start 2024-03-17T23:55:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.839+0000 7f0922c00640 1 -- start start 2024-03-17T23:55:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.839+0000 7f0922c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f091c0761a0 0x7f091c19e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.839+0000 7f0922c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f091c0773e0 0x7f091c19e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.839+0000 7f0922c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c077d80 0x7f091c1a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.839+0000 7f0922c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f091c113970 con 0x7f091c077d80 2024-03-17T23:55:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.839+0000 7f0922c00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1137f0 con 0x7f091c0773e0 2024-03-17T23:55:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.839+0000 7f0922c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f091c113af0 con 0x7f091c0761a0 2024-03-17T23:55:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.839+0000 7f0920e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f091c0773e0 0x7f091c19e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.839+0000 7f0921800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f091c0761a0 0x7f091c19e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.840+0000 7f0920e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f091c0773e0 0x7f091c19e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:52946/0 (socket says 172.21.15.12:52946) 2024-03-17T23:55:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.840+0000 7f0920e00640 1 -- 172.21.15.12:0/2142571036 learned_addr learned my addr 172.21.15.12:0/2142571036 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.840+0000 7f0921800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f091c0761a0 0x7f091c19e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:45182/0 (socket says 172.21.15.12:45182) 2024-03-17T23:55:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.840+0000 7f0922200640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c077d80 0x7f091c1a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:16.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.840+0000 7f0920e00640 1 -- 172.21.15.12:0/2142571036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f091c0761a0 msgr2=0x7f091c19e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.840+0000 7f0920e00640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f091c0761a0 0x7f091c19e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.840+0000 7f0920e00640 1 -- 172.21.15.12:0/2142571036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c077d80 msgr2=0x7f091c1a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.840+0000 7f0920e00640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c077d80 0x7f091c1a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.840+0000 7f0920e00640 1 -- 172.21.15.12:0/2142571036 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f091c1a5c70 con 0x7f091c0773e0 2024-03-17T23:55:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.840+0000 7f0922200640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c077d80 0x7f091c1a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:55:16.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.840+0000 7f0921800640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f091c0761a0 0x7f091c19e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:55:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.840+0000 7f0920e00640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f091c0773e0 0x7f091c19e590 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f090c0349e0 tx=0x7f090c032450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.841+0000 7f091a000640 1 -- 172.21.15.12:0/2142571036 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c041030 con 0x7f091c0773e0 2024-03-17T23:55:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.841+0000 7f091a000640 1 -- 172.21.15.12:0/2142571036 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f090c02c510 con 0x7f091c0773e0 2024-03-17T23:55:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.841+0000 7f091a000640 1 -- 172.21.15.12:0/2142571036 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c039040 con 0x7f091c0773e0 2024-03-17T23:55:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.841+0000 7f0922c00640 1 -- 172.21.15.12:0/2142571036 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f091c1a5f00 con 0x7f091c0773e0 2024-03-17T23:55:16.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.841+0000 7f0922c00640 1 -- 172.21.15.12:0/2142571036 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f091c1a6370 con 0x7f091c0773e0 2024-03-17T23:55:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.842+0000 7f0922c00640 1 -- 172.21.15.12:0/2142571036 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f091c110490 con 0x7f091c0773e0 2024-03-17T23:55:16.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.845+0000 7f091a000640 1 -- 172.21.15.12:0/2142571036 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f090c044070 con 0x7f091c0773e0 2024-03-17T23:55:16.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.846+0000 7f091a000640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f09040a2880 0x7f09040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:16.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.846+0000 7f091a000640 1 -- 172.21.15.12:0/2142571036 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f090c0f0250 con 0x7f091c0773e0 2024-03-17T23:55:16.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.846+0000 7f0921800640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f09040a2880 0x7f09040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:16.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.847+0000 7f0921800640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f09040a2880 0x7f09040a4d40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0910009fc0 tx=0x7f0910005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:16.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:16.854+0000 7f091a000640 1 -- 172.21.15.12:0/2142571036 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f090c0bd050 con 0x7f091c0773e0 2024-03-17T23:55:17.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.057+0000 7f0922c00640 1 -- 172.21.15.12:0/2142571036 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f091c0623e0 con 0x7f091c0773e0 2024-03-17T23:55:17.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.057+0000 7f091a000640 1 -- 172.21.15.12:0/2142571036 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f090c0bc950 con 0x7f091c0773e0 2024-03-17T23:55:17.058 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-17T23:55:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.060+0000 7f0922c00640 1 -- 172.21.15.12:0/2142571036 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f09040a2880 msgr2=0x7f09040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.060+0000 7f0922c00640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f09040a2880 0x7f09040a4d40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0910009fc0 tx=0x7f0910005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.060+0000 7f0922c00640 1 -- 172.21.15.12:0/2142571036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f091c0773e0 msgr2=0x7f091c19e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.060+0000 7f0922c00640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f091c0773e0 0x7f091c19e590 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f090c0349e0 tx=0x7f090c032450 comp rx=0 tx=0).stop 2024-03-17T23:55:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.060+0000 7f0922c00640 1 -- 172.21.15.12:0/2142571036 shutdown_connections 2024-03-17T23:55:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.060+0000 7f0922c00640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f09040a2880 0x7f09040a4d40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.060+0000 7f0922c00640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f091c077d80 0x7f091c1a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.060+0000 7f0922c00640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f091c0773e0 0x7f091c19e590 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.060+0000 7f0922c00640 1 --2- 172.21.15.12:0/2142571036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f091c0761a0 0x7f091c19e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.060+0000 7f0922c00640 1 -- 172.21.15.12:0/2142571036 >> 172.21.15.12:0/2142571036 conn(0x7f091c0fee50 msgr2=0x7f091c100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.061+0000 7f0922c00640 1 -- 172.21.15.12:0/2142571036 shutdown_connections 2024-03-17T23:55:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.061+0000 7f0922c00640 1 -- 172.21.15.12:0/2142571036 wait complete. 2024-03-17T23:55:17.079 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:55:17.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:16 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4262176532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:55:17.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:16 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3929776702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:55:17.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.285+0000 7fc507e00640 1 -- 172.21.15.12:0/3925894657 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc508104a70 msgr2=0x7fc508106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:17.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.285+0000 7fc507e00640 1 --2- 172.21.15.12:0/3925894657 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc508104a70 0x7fc508106e60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0009ca0 tx=0x7fc4f002c100 comp rx=0 tx=0).stop 2024-03-17T23:55:17.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.285+0000 7fc507e00640 1 -- 172.21.15.12:0/3925894657 shutdown_connections 2024-03-17T23:55:17.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.285+0000 7fc507e00640 1 --2- 172.21.15.12:0/3925894657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5081073a0 0x7fc5081097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.285+0000 7fc507e00640 1 --2- 172.21.15.12:0/3925894657 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc508104a70 0x7fc508106e60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.285+0000 7fc507e00640 1 --2- 172.21.15.12:0/3925894657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc508102140 0x7fc508104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.285+0000 7fc507e00640 1 -- 172.21.15.12:0/3925894657 >> 172.21.15.12:0/3925894657 conn(0x7fc5080fbb50 msgr2=0x7fc5080fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:17.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.285+0000 7fc507e00640 1 -- 172.21.15.12:0/3925894657 shutdown_connections 2024-03-17T23:55:17.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.285+0000 7fc507e00640 1 -- 172.21.15.12:0/3925894657 wait complete. 2024-03-17T23:55:17.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.286+0000 7fc507e00640 1 Processor -- start 2024-03-17T23:55:17.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.287+0000 7fc507e00640 1 -- start start 2024-03-17T23:55:17.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.287+0000 7fc507e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc508102140 0x7fc508199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:17.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.287+0000 7fc507e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc508104a70 0x7fc50819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:17.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.287+0000 7fc507e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5081073a0 0x7fc5081a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:17.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.287+0000 7fc507e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc50810d340 con 0x7fc508102140 2024-03-17T23:55:17.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.287+0000 7fc507e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc50810d1c0 con 0x7fc508104a70 2024-03-17T23:55:17.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.287+0000 7fc507e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc50810d4c0 con 0x7fc5081073a0 2024-03-17T23:55:17.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.287+0000 7fc507400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5081073a0 0x7fc5081a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:17.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.287+0000 7fc506000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc508104a70 0x7fc50819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:17.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.287+0000 7fc506a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc508102140 0x7fc508199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:17.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.287+0000 7fc507400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5081073a0 0x7fc5081a1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:45200/0 (socket says 172.21.15.12:45200) 2024-03-17T23:55:17.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.288+0000 7fc507400640 1 -- 172.21.15.12:0/250309200 learned_addr learned my addr 172.21.15.12:0/250309200 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:17.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.288+0000 7fc506000640 1 -- 172.21.15.12:0/250309200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5081073a0 msgr2=0x7fc5081a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:17.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.288+0000 7fc506000640 1 --2- 172.21.15.12:0/250309200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5081073a0 0x7fc5081a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.288+0000 7fc506000640 1 -- 172.21.15.12:0/250309200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc508102140 msgr2=0x7fc508199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:17.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.288+0000 7fc506000640 1 --2- 172.21.15.12:0/250309200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc508102140 0x7fc508199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.288+0000 7fc506000640 1 -- 172.21.15.12:0/250309200 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5081a17f0 con 0x7fc508104a70 2024-03-17T23:55:17.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.288+0000 7fc507400640 1 --2- 172.21.15.12:0/250309200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5081073a0 0x7fc5081a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:55:17.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.288+0000 7fc506000640 1 --2- 172.21.15.12:0/250309200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc508104a70 0x7fc50819a110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0037040 tx=0x7fc4f0035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:17.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.289+0000 7fc4ff400640 1 -- 172.21.15.12:0/250309200 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0039040 con 0x7fc508104a70 2024-03-17T23:55:17.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.289+0000 7fc4ff400640 1 -- 172.21.15.12:0/250309200 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4f0003970 con 0x7fc508104a70 2024-03-17T23:55:17.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.289+0000 7fc4ff400640 1 -- 172.21.15.12:0/250309200 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f00043c0 con 0x7fc508104a70 2024-03-17T23:55:17.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.289+0000 7fc507e00640 1 -- 172.21.15.12:0/250309200 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5081a1a80 con 0x7fc508104a70 2024-03-17T23:55:17.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.289+0000 7fc507e00640 1 -- 172.21.15.12:0/250309200 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5081a1ef0 con 0x7fc508104a70 2024-03-17T23:55:17.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.290+0000 7fc507e00640 1 -- 172.21.15.12:0/250309200 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc508108ef0 con 0x7fc508104a70 2024-03-17T23:55:17.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.293+0000 7fc4ff400640 1 -- 172.21.15.12:0/250309200 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc4f0046030 con 0x7fc508104a70 2024-03-17T23:55:17.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.294+0000 7fc4ff400640 1 --2- 172.21.15.12:0/250309200 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc4ec0a2800 0x7fc4ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:17.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.294+0000 7fc4ff400640 1 -- 172.21.15.12:0/250309200 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc4f00f06a0 con 0x7fc508104a70 2024-03-17T23:55:17.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.294+0000 7fc506a00640 1 --2- 172.21.15.12:0/250309200 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc4ec0a2800 0x7fc4ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:17.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:16 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/4262176532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:55:17.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:16 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3929776702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:55:17.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.295+0000 7fc506a00640 1 --2- 172.21.15.12:0/250309200 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc4ec0a2800 0x7fc4ec0a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009bd0 tx=0x7fc4f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:17.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.303+0000 7fc4ff400640 1 -- 172.21.15.12:0/250309200 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc4f00bd530 con 0x7fc508104a70 2024-03-17T23:55:17.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:16 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/4262176532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:55:17.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:16 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3929776702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:55:17.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.511+0000 7fc507e00640 1 -- 172.21.15.12:0/250309200 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc50802d1b0 con 0x7fc508104a70 2024-03-17T23:55:17.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.512+0000 7fc4ff400640 1 -- 172.21.15.12:0/250309200 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+590 (secure 0 0 0) 0x7fc4f00bce30 con 0x7fc508104a70 2024-03-17T23:55:17.513 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/2526360059 2024-03-18T23:46:53.955890+0000 2024-03-17T23:55:17.513 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/2336882461 2024-03-18T23:46:53.955890+0000 2024-03-17T23:55:17.513 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/1563339918 2024-03-18T23:46:19.706199+0000 2024-03-17T23:55:17.513 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1976882257 2024-03-18T23:46:53.955890+0000 2024-03-17T23:55:17.513 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/501542164 2024-03-18T23:46:53.955890+0000 2024-03-17T23:55:17.513 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/654354506 2024-03-18T23:46:19.706199+0000 2024-03-17T23:55:17.513 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/1563339918 2024-03-18T23:46:19.706199+0000 2024-03-17T23:55:17.513 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1974593432 2024-03-18T23:46:19.706199+0000 2024-03-17T23:55:17.513 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/2526360059 2024-03-18T23:46:53.955890+0000 2024-03-17T23:55:17.513 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1690054132 2024-03-18T23:46:19.706199+0000 2024-03-17T23:55:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:listed 10 entries 2024-03-17T23:55:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.514+0000 7fc507e00640 1 -- 172.21.15.12:0/250309200 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc4ec0a2800 msgr2=0x7fc4ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.514+0000 7fc507e00640 1 --2- 172.21.15.12:0/250309200 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc4ec0a2800 0x7fc4ec0a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009bd0 tx=0x7fc4f8005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.515+0000 7fc507e00640 1 -- 172.21.15.12:0/250309200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc508104a70 msgr2=0x7fc50819a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.515+0000 7fc507e00640 1 --2- 172.21.15.12:0/250309200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc508104a70 0x7fc50819a110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0037040 tx=0x7fc4f0035d10 comp rx=0 tx=0).stop 2024-03-17T23:55:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.515+0000 7fc507e00640 1 -- 172.21.15.12:0/250309200 shutdown_connections 2024-03-17T23:55:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.515+0000 7fc507e00640 1 --2- 172.21.15.12:0/250309200 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc4ec0a2800 0x7fc4ec0a4cc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.515+0000 7fc507e00640 1 --2- 172.21.15.12:0/250309200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5081073a0 0x7fc5081a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.515+0000 7fc507e00640 1 --2- 172.21.15.12:0/250309200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc508104a70 0x7fc50819a110 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.515+0000 7fc507e00640 1 --2- 172.21.15.12:0/250309200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc508102140 0x7fc508199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.515+0000 7fc507e00640 1 -- 172.21.15.12:0/250309200 >> 172.21.15.12:0/250309200 conn(0x7fc5080fbb50 msgr2=0x7fc508101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.515+0000 7fc507e00640 1 -- 172.21.15.12:0/250309200 shutdown_connections 2024-03-17T23:55:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.515+0000 7fc507e00640 1 -- 172.21.15.12:0/250309200 wait complete. 2024-03-17T23:55:17.533 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:55:17.533 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:55:17.581 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:55:17.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.844+0000 7fa081200640 1 -- 172.21.15.12:0/1340145421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa07c101810 msgr2=0x7fa07c101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:17.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.844+0000 7fa081200640 1 --2- 172.21.15.12:0/1340145421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa07c101810 0x7fa07c101c30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa06c009900 tx=0x7fa06c02bc40 comp rx=0 tx=0).stop 2024-03-17T23:55:17.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.844+0000 7fa081200640 1 -- 172.21.15.12:0/1340145421 shutdown_connections 2024-03-17T23:55:17.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.844+0000 7fa081200640 1 --2- 172.21.15.12:0/1340145421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa07c1033c0 0x7fa07c109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.844+0000 7fa081200640 1 --2- 172.21.15.12:0/1340145421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa07c102a00 0x7fa07c102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.844+0000 7fa081200640 1 --2- 172.21.15.12:0/1340145421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa07c101810 0x7fa07c101c30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.844+0000 7fa081200640 1 -- 172.21.15.12:0/1340145421 >> 172.21.15.12:0/1340145421 conn(0x7fa07c0fcd80 msgr2=0x7fa07c0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:17.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.844+0000 7fa081200640 1 -- 172.21.15.12:0/1340145421 shutdown_connections 2024-03-17T23:55:17.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.844+0000 7fa081200640 1 -- 172.21.15.12:0/1340145421 wait complete. 2024-03-17T23:55:17.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.845+0000 7fa081200640 1 Processor -- start 2024-03-17T23:55:17.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.845+0000 7fa081200640 1 -- start start 2024-03-17T23:55:17.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.845+0000 7fa081200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa07c101810 0x7fa07c075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:17.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.845+0000 7fa081200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa07c102a00 0x7fa07c073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:17.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.845+0000 7fa081200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa07c1033c0 0x7fa07c074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:17.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.845+0000 7fa081200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa07c10d460 con 0x7fa07c102a00 2024-03-17T23:55:17.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.845+0000 7fa081200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa07c10d2e0 con 0x7fa07c1033c0 2024-03-17T23:55:17.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.845+0000 7fa081200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa07c10d5e0 con 0x7fa07c101810 2024-03-17T23:55:17.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.846+0000 7fa07a800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa07c1033c0 0x7fa07c074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:17.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.846+0000 7fa079400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa07c102a00 0x7fa07c073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:17.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.846+0000 7fa07a800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa07c1033c0 0x7fa07c074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:52984/0 (socket says 172.21.15.12:52984) 2024-03-17T23:55:17.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.846+0000 7fa07a800640 1 -- 172.21.15.12:0/95410430 learned_addr learned my addr 172.21.15.12:0/95410430 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:17.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.846+0000 7fa079e00640 1 --2- 172.21.15.12:0/95410430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa07c101810 0x7fa07c075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:17.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.846+0000 7fa07a800640 1 -- 172.21.15.12:0/95410430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa07c101810 msgr2=0x7fa07c075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:17.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.846+0000 7fa07a800640 1 --2- 172.21.15.12:0/95410430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa07c101810 0x7fa07c075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.846+0000 7fa07a800640 1 -- 172.21.15.12:0/95410430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa07c102a00 msgr2=0x7fa07c073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:17.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.846+0000 7fa07a800640 1 --2- 172.21.15.12:0/95410430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa07c102a00 0x7fa07c073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:17.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.846+0000 7fa07a800640 1 -- 172.21.15.12:0/95410430 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa07c0749a0 con 0x7fa07c1033c0 2024-03-17T23:55:17.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.847+0000 7fa07a800640 1 --2- 172.21.15.12:0/95410430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa07c1033c0 0x7fa07c074110 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa07400d750 tx=0x7fa07400dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:17.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.847+0000 7fa072a00640 1 -- 172.21.15.12:0/95410430 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa074017070 con 0x7fa07c1033c0 2024-03-17T23:55:17.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.847+0000 7fa072a00640 1 -- 172.21.15.12:0/95410430 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa074005500 con 0x7fa07c1033c0 2024-03-17T23:55:17.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.847+0000 7fa072a00640 1 -- 172.21.15.12:0/95410430 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa074004d90 con 0x7fa07c1033c0 2024-03-17T23:55:17.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.847+0000 7fa081200640 1 -- 172.21.15.12:0/95410430 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa07c1a1c30 con 0x7fa07c1033c0 2024-03-17T23:55:17.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.847+0000 7fa081200640 1 -- 172.21.15.12:0/95410430 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa07c1a2130 con 0x7fa07c1033c0 2024-03-17T23:55:17.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.848+0000 7fa081200640 1 -- 172.21.15.12:0/95410430 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa07c108b30 con 0x7fa07c1033c0 2024-03-17T23:55:17.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.850+0000 7fa072a00640 1 -- 172.21.15.12:0/95410430 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa074004f30 con 0x7fa07c1033c0 2024-03-17T23:55:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.851+0000 7fa072a00640 1 --2- 172.21.15.12:0/95410430 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa05c0a2800 0x7fa05c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.851+0000 7fa072a00640 1 -- 172.21.15.12:0/95410430 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa0740ce660 con 0x7fa07c1033c0 2024-03-17T23:55:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.852+0000 7fa079e00640 1 --2- 172.21.15.12:0/95410430 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa05c0a2800 0x7fa05c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.852+0000 7fa079e00640 1 --2- 172.21.15.12:0/95410430 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa05c0a2800 0x7fa05c0a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa06c000c00 tx=0x7fa06c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:17.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:17.859+0000 7fa072a00640 1 -- 172.21.15.12:0/95410430 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa07409ae10 con 0x7fa07c1033c0 2024-03-17T23:55:18.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.068+0000 7fa081200640 1 -- 172.21.15.12:0/95410430 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa07c062480 con 0x7fa07c1033c0 2024-03-17T23:55:18.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.070+0000 7fa072a00640 1 -- 172.21.15.12:0/95410430 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24044 (secure 0 0 0) 0x7fa07409eeb0 con 0x7fa07c1033c0 2024-03-17T23:55:18.071 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:18.072 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":90,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:55:02.337441+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:55:18.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.073+0000 7fa081200640 1 -- 172.21.15.12:0/95410430 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa05c0a2800 msgr2=0x7fa05c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:18.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.073+0000 7fa081200640 1 --2- 172.21.15.12:0/95410430 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa05c0a2800 0x7fa05c0a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa06c000c00 tx=0x7fa06c005fd0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.073+0000 7fa081200640 1 -- 172.21.15.12:0/95410430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa07c1033c0 msgr2=0x7fa07c074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:18.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.073+0000 7fa081200640 1 --2- 172.21.15.12:0/95410430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa07c1033c0 0x7fa07c074110 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa07400d750 tx=0x7fa07400dc10 comp rx=0 tx=0).stop 2024-03-17T23:55:18.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.073+0000 7fa081200640 1 -- 172.21.15.12:0/95410430 shutdown_connections 2024-03-17T23:55:18.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.073+0000 7fa081200640 1 --2- 172.21.15.12:0/95410430 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa05c0a2800 0x7fa05c0a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.073+0000 7fa081200640 1 --2- 172.21.15.12:0/95410430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa07c1033c0 0x7fa07c074110 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.073+0000 7fa081200640 1 --2- 172.21.15.12:0/95410430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa07c102a00 0x7fa07c073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.073+0000 7fa081200640 1 --2- 172.21.15.12:0/95410430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa07c101810 0x7fa07c075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.073+0000 7fa081200640 1 -- 172.21.15.12:0/95410430 >> 172.21.15.12:0/95410430 conn(0x7fa07c0fcd80 msgr2=0x7fa07c105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:18.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.073+0000 7fa081200640 1 -- 172.21.15.12:0/95410430 shutdown_connections 2024-03-17T23:55:18.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.073+0000 7fa081200640 1 -- 172.21.15.12:0/95410430 wait complete. 2024-03-17T23:55:18.094 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:49:38.640282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:54:54.540345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:54:55.803926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:55:00.355066+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:55:18.094 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:55:18.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:17 smithi012 ceph-mon[31541]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-17T23:55:18.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:17 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2142571036' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:55:18.177 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:17 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/250309200' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-17T23:55:18.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:17 smithi079 ceph-mon[36241]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-17T23:55:18.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:17 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2142571036' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:55:18.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:17 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/250309200' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-17T23:55:18.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.315+0000 7fa322800640 1 -- 172.21.15.12:0/2806973322 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa31c103900 msgr2=0x7fa31c103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:18.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.315+0000 7fa322800640 1 --2- 172.21.15.12:0/2806973322 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa31c103900 0x7fa31c103d20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa310009900 tx=0x7fa31002bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:18.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.315+0000 7fa322800640 1 -- 172.21.15.12:0/2806973322 shutdown_connections 2024-03-17T23:55:18.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.315+0000 7fa322800640 1 --2- 172.21.15.12:0/2806973322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa31c1054b0 0x7fa31c10bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.315+0000 7fa322800640 1 --2- 172.21.15.12:0/2806973322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa31c104af0 0x7fa31c104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.315+0000 7fa322800640 1 --2- 172.21.15.12:0/2806973322 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa31c103900 0x7fa31c103d20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.315+0000 7fa322800640 1 -- 172.21.15.12:0/2806973322 >> 172.21.15.12:0/2806973322 conn(0x7fa31c0fee90 msgr2=0x7fa31c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:18.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.315+0000 7fa322800640 1 -- 172.21.15.12:0/2806973322 shutdown_connections 2024-03-17T23:55:18.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.315+0000 7fa322800640 1 -- 172.21.15.12:0/2806973322 wait complete. 2024-03-17T23:55:18.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.316+0000 7fa322800640 1 Processor -- start 2024-03-17T23:55:18.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.317+0000 7fa322800640 1 -- start start 2024-03-17T23:55:18.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.317+0000 7fa322800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa31c103900 0x7fa31c199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:18.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.317+0000 7fa322800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa31c104af0 0x7fa31c19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:18.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.317+0000 7fa322800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa31c1054b0 0x7fa31c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:18.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.317+0000 7fa322800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c10f5a0 con 0x7fa31c103900 2024-03-17T23:55:18.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.317+0000 7fa322800640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c10f420 con 0x7fa31c104af0 2024-03-17T23:55:18.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.317+0000 7fa322800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c10f720 con 0x7fa31c1054b0 2024-03-17T23:55:18.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.317+0000 7fa320a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa31c104af0 0x7fa31c19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:18.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.317+0000 7fa320a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa31c104af0 0x7fa31c19a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:52998/0 (socket says 172.21.15.12:52998) 2024-03-17T23:55:18.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.317+0000 7fa320a00640 1 -- 172.21.15.12:0/205850879 learned_addr learned my addr 172.21.15.12:0/205850879 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:18.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.317+0000 7fa321e00640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa31c1054b0 0x7fa31c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:18.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.317+0000 7fa321400640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa31c103900 0x7fa31c199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:18.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.318+0000 7fa320a00640 1 -- 172.21.15.12:0/205850879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa31c1054b0 msgr2=0x7fa31c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:18.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.318+0000 7fa320a00640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa31c1054b0 0x7fa31c1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.318+0000 7fa320a00640 1 -- 172.21.15.12:0/205850879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa31c103900 msgr2=0x7fa31c199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:18.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.318+0000 7fa320a00640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa31c103900 0x7fa31c199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.318+0000 7fa320a00640 1 -- 172.21.15.12:0/205850879 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31c1a1810 con 0x7fa31c104af0 2024-03-17T23:55:18.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.318+0000 7fa321e00640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa31c1054b0 0x7fa31c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:55:18.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.318+0000 7fa320a00640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa31c104af0 0x7fa31c19a130 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa30800e9f0 tx=0x7fa30800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:18.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.319+0000 7fa31a000640 1 -- 172.21.15.12:0/205850879 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30800cd80 con 0x7fa31c104af0 2024-03-17T23:55:18.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.319+0000 7fa31a000640 1 -- 172.21.15.12:0/205850879 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa308010910 con 0x7fa31c104af0 2024-03-17T23:55:18.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.319+0000 7fa31a000640 1 -- 172.21.15.12:0/205850879 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa308018980 con 0x7fa31c104af0 2024-03-17T23:55:18.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.319+0000 7fa322800640 1 -- 172.21.15.12:0/205850879 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa31c1a1b00 con 0x7fa31c104af0 2024-03-17T23:55:18.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.319+0000 7fa322800640 1 -- 172.21.15.12:0/205850879 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa31c1a2000 con 0x7fa31c104af0 2024-03-17T23:55:18.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.320+0000 7fa322800640 1 -- 172.21.15.12:0/205850879 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa31c10ac20 con 0x7fa31c104af0 2024-03-17T23:55:18.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.322+0000 7fa31a000640 1 -- 172.21.15.12:0/205850879 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa308010ac0 con 0x7fa31c104af0 2024-03-17T23:55:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.324+0000 7fa31a000640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3040a2880 0x7fa3040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.324+0000 7fa31a000640 1 -- 172.21.15.12:0/205850879 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa3080cf1e0 con 0x7fa31c104af0 2024-03-17T23:55:18.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.324+0000 7fa321400640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3040a2880 0x7fa3040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:18.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.325+0000 7fa321400640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3040a2880 0x7fa3040a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa31002c170 tx=0x7fa31002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:18.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.325+0000 7fa31a000640 1 -- 172.21.15.12:0/205850879 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa30809bfe0 con 0x7fa31c104af0 2024-03-17T23:55:18.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:17 smithi191 ceph-mon[36708]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-17T23:55:18.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:17 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2142571036' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:55:18.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:17 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/250309200' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-17T23:55:18.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.518+0000 7fa322800640 1 -- 172.21.15.12:0/205850879 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa31c02d400 con 0x7fa31c104af0 2024-03-17T23:55:18.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.518+0000 7fa31a000640 1 -- 172.21.15.12:0/205850879 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fa30809b8e0 con 0x7fa31c104af0 2024-03-17T23:55:18.519 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-17T23:55:18.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.521+0000 7fa322800640 1 -- 172.21.15.12:0/205850879 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3040a2880 msgr2=0x7fa3040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:18.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.521+0000 7fa322800640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3040a2880 0x7fa3040a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa31002c170 tx=0x7fa31002df90 comp rx=0 tx=0).stop 2024-03-17T23:55:18.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.521+0000 7fa322800640 1 -- 172.21.15.12:0/205850879 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa31c104af0 msgr2=0x7fa31c19a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:18.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.521+0000 7fa322800640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa31c104af0 0x7fa31c19a130 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa30800e9f0 tx=0x7fa30800eeb0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.521+0000 7fa322800640 1 -- 172.21.15.12:0/205850879 shutdown_connections 2024-03-17T23:55:18.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.521+0000 7fa322800640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa3040a2880 0x7fa3040a4d40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.521+0000 7fa322800640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa31c1054b0 0x7fa31c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.521+0000 7fa322800640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa31c104af0 0x7fa31c19a130 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.521+0000 7fa322800640 1 --2- 172.21.15.12:0/205850879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa31c103900 0x7fa31c199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.521+0000 7fa322800640 1 -- 172.21.15.12:0/205850879 >> 172.21.15.12:0/205850879 conn(0x7fa31c0fee90 msgr2=0x7fa31c107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:18.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.521+0000 7fa322800640 1 -- 172.21.15.12:0/205850879 shutdown_connections 2024-03-17T23:55:18.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.522+0000 7fa322800640 1 -- 172.21.15.12:0/205850879 wait complete. 2024-03-17T23:55:18.542 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:55:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.766+0000 7f810b200640 1 -- 172.21.15.12:0/1976394743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f810c1016c0 msgr2=0x7f810c101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.766+0000 7f810b200640 1 --2- 172.21.15.12:0/1976394743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f810c1016c0 0x7f810c101ae0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f80fc009900 tx=0x7f80fc02bc40 comp rx=0 tx=0).stop 2024-03-17T23:55:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.766+0000 7f810b200640 1 -- 172.21.15.12:0/1976394743 shutdown_connections 2024-03-17T23:55:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.766+0000 7f810b200640 1 --2- 172.21.15.12:0/1976394743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f810c103270 0x7f810c109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.766+0000 7f810b200640 1 --2- 172.21.15.12:0/1976394743 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f810c1028b0 0x7f810c102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.766+0000 7f810b200640 1 --2- 172.21.15.12:0/1976394743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f810c1016c0 0x7f810c101ae0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.766+0000 7f810b200640 1 -- 172.21.15.12:0/1976394743 >> 172.21.15.12:0/1976394743 conn(0x7f810c0fcc50 msgr2=0x7f810c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:18.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.766+0000 7f810b200640 1 -- 172.21.15.12:0/1976394743 shutdown_connections 2024-03-17T23:55:18.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.766+0000 7f810b200640 1 -- 172.21.15.12:0/1976394743 wait complete. 2024-03-17T23:55:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.767+0000 7f810b200640 1 Processor -- start 2024-03-17T23:55:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.767+0000 7f810b200640 1 -- start start 2024-03-17T23:55:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f810b200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f810c1016c0 0x7f810c1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f810b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f810c1028b0 0x7f810c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f810b200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f810c103270 0x7f810c19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f810b200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f810c10d340 con 0x7f810c103270 2024-03-17T23:55:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f810b200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f810c10d1c0 con 0x7f810c1016c0 2024-03-17T23:55:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f810b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f810c10d4c0 con 0x7f810c1028b0 2024-03-17T23:55:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f8109e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f810c1016c0 0x7f810c1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f8109e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f810c1016c0 0x7f810c1979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:53016/0 (socket says 172.21.15.12:53016) 2024-03-17T23:55:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f8109e00640 1 -- 172.21.15.12:0/4274803250 learned_addr learned my addr 172.21.15.12:0/4274803250 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f810a800640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f810c103270 0x7f810c19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f8109400640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f810c1028b0 0x7f810c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f8109e00640 1 -- 172.21.15.12:0/4274803250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f810c1028b0 msgr2=0x7f810c197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f8109e00640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f810c1028b0 0x7f810c197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f8109e00640 1 -- 172.21.15.12:0/4274803250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f810c103270 msgr2=0x7f810c19ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f8109e00640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f810c103270 0x7f810c19ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.768+0000 7f8109e00640 1 -- 172.21.15.12:0/4274803250 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f810c19f600 con 0x7f810c1016c0 2024-03-17T23:55:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.769+0000 7f8109e00640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f810c1016c0 0x7f810c1979e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f80fc000c00 tx=0x7f80fc005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.769+0000 7f8102a00640 1 -- 172.21.15.12:0/4274803250 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc041070 con 0x7f810c1016c0 2024-03-17T23:55:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.769+0000 7f8102a00640 1 -- 172.21.15.12:0/4274803250 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80fc0044d0 con 0x7f810c1016c0 2024-03-17T23:55:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.769+0000 7f810b200640 1 -- 172.21.15.12:0/4274803250 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f810c19f890 con 0x7f810c1016c0 2024-03-17T23:55:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.769+0000 7f8102a00640 1 -- 172.21.15.12:0/4274803250 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc034440 con 0x7f810c1016c0 2024-03-17T23:55:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.769+0000 7f810b200640 1 -- 172.21.15.12:0/4274803250 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f810c19fdd0 con 0x7f810c1016c0 2024-03-17T23:55:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.770+0000 7f810b200640 1 -- 172.21.15.12:0/4274803250 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f810c1089e0 con 0x7f810c1016c0 2024-03-17T23:55:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.772+0000 7f8102a00640 1 -- 172.21.15.12:0/4274803250 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f80fc044070 con 0x7f810c1016c0 2024-03-17T23:55:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.774+0000 7f8102a00640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f80f00a2880 0x7f80f00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.774+0000 7f8102a00640 1 -- 172.21.15.12:0/4274803250 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f80fc0f0840 con 0x7f810c1016c0 2024-03-17T23:55:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.774+0000 7f8109400640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f80f00a2880 0x7f80f00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.775+0000 7f8109400640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f80f00a2880 0x7f80f00a4d40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f80f80095f0 tx=0x7f80f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.778+0000 7f8102a00640 1 -- 172.21.15.12:0/4274803250 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f80fc0bcff0 con 0x7f810c1016c0 2024-03-17T23:55:18.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.985+0000 7f810b200640 1 -- 172.21.15.12:0/4274803250 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f810c0623e0 con 0x7f810c1016c0 2024-03-17T23:55:18.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.985+0000 7f8102a00640 1 -- 172.21.15.12:0/4274803250 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f80fc0c1090 con 0x7f810c1016c0 2024-03-17T23:55:18.986 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-17T23:55:18.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.988+0000 7f810b200640 1 -- 172.21.15.12:0/4274803250 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f80f00a2880 msgr2=0x7f80f00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:18.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.988+0000 7f810b200640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f80f00a2880 0x7f80f00a4d40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f80f80095f0 tx=0x7f80f8005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:18.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.988+0000 7f810b200640 1 -- 172.21.15.12:0/4274803250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f810c1016c0 msgr2=0x7f810c1979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:18.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.988+0000 7f810b200640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f810c1016c0 0x7f810c1979e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f80fc000c00 tx=0x7f80fc005d50 comp rx=0 tx=0).stop 2024-03-17T23:55:18.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.988+0000 7f810b200640 1 -- 172.21.15.12:0/4274803250 shutdown_connections 2024-03-17T23:55:18.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.988+0000 7f810b200640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f80f00a2880 0x7f80f00a4d40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.988+0000 7f810b200640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f810c103270 0x7f810c19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.988+0000 7f810b200640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f810c1028b0 0x7f810c197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.988+0000 7f810b200640 1 --2- 172.21.15.12:0/4274803250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f810c1016c0 0x7f810c1979e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:18.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.988+0000 7f810b200640 1 -- 172.21.15.12:0/4274803250 >> 172.21.15.12:0/4274803250 conn(0x7f810c0fcc50 msgr2=0x7f810c105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:18.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.988+0000 7f810b200640 1 -- 172.21.15.12:0/4274803250 shutdown_connections 2024-03-17T23:55:18.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:18.988+0000 7f810b200640 1 -- 172.21.15.12:0/4274803250 wait complete. 2024-03-17T23:55:19.008 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:55:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.226+0000 7f0bf9200640 1 -- 172.21.15.12:0/852479961 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0bf41055b0 msgr2=0x7f0bf410be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.226+0000 7f0bf9200640 1 --2- 172.21.15.12:0/852479961 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0bf41055b0 0x7f0bf410be40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0bec009be0 tx=0x7f0bec02bef0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.226+0000 7f0bf9200640 1 -- 172.21.15.12:0/852479961 shutdown_connections 2024-03-17T23:55:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.226+0000 7f0bf9200640 1 --2- 172.21.15.12:0/852479961 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0bf41055b0 0x7f0bf410be40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.226+0000 7f0bf9200640 1 --2- 172.21.15.12:0/852479961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bf4104bf0 0x7f0bf4105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.226+0000 7f0bf9200640 1 --2- 172.21.15.12:0/852479961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bf4103a00 0x7f0bf4103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.226+0000 7f0bf9200640 1 -- 172.21.15.12:0/852479961 >> 172.21.15.12:0/852479961 conn(0x7f0bf40fef90 msgr2=0x7f0bf41013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:19.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.226+0000 7f0bf9200640 1 -- 172.21.15.12:0/852479961 shutdown_connections 2024-03-17T23:55:19.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.226+0000 7f0bf9200640 1 -- 172.21.15.12:0/852479961 wait complete. 2024-03-17T23:55:19.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.227+0000 7f0bf9200640 1 Processor -- start 2024-03-17T23:55:19.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.227+0000 7f0bf9200640 1 -- start start 2024-03-17T23:55:19.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0bf9200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bf4103a00 0x7f0bf4199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:19.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0bf9200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0bf4104bf0 0x7f0bf419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:19.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0bf9200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bf41055b0 0x7f0bf41a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:19.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0bf9200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf410f6d0 con 0x7f0bf41055b0 2024-03-17T23:55:19.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0bf9200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf410f550 con 0x7f0bf4104bf0 2024-03-17T23:55:19.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0bf9200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf410f850 con 0x7f0bf4103a00 2024-03-17T23:55:19.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0be9400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0bf4104bf0 0x7f0bf419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:19.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0be9400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0bf4104bf0 0x7f0bf419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:53032/0 (socket says 172.21.15.12:53032) 2024-03-17T23:55:19.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0be9400640 1 -- 172.21.15.12:0/1805194177 learned_addr learned my addr 172.21.15.12:0/1805194177 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:19.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0bf2800640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bf41055b0 0x7f0bf41a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:19.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0bf1e00640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bf4103a00 0x7f0bf4199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:19.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0be9400640 1 -- 172.21.15.12:0/1805194177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bf4103a00 msgr2=0x7f0bf4199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:19.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0be9400640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bf4103a00 0x7f0bf4199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0be9400640 1 -- 172.21.15.12:0/1805194177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bf41055b0 msgr2=0x7f0bf41a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:19.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0be9400640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bf41055b0 0x7f0bf41a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.228+0000 7f0be9400640 1 -- 172.21.15.12:0/1805194177 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf41a18d0 con 0x7f0bf4104bf0 2024-03-17T23:55:19.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.229+0000 7f0be9400640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0bf4104bf0 0x7f0bf419a1d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc00e9f0 tx=0x7f0bdc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:19.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.229+0000 7f0beb400640 1 -- 172.21.15.12:0/1805194177 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc00cd80 con 0x7f0bf4104bf0 2024-03-17T23:55:19.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.229+0000 7f0beb400640 1 -- 172.21.15.12:0/1805194177 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bdc005500 con 0x7f0bf4104bf0 2024-03-17T23:55:19.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.229+0000 7f0beb400640 1 -- 172.21.15.12:0/1805194177 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc010690 con 0x7f0bf4104bf0 2024-03-17T23:55:19.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.229+0000 7f0bf9200640 1 -- 172.21.15.12:0/1805194177 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf41a1bc0 con 0x7f0bf4104bf0 2024-03-17T23:55:19.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.229+0000 7f0bf9200640 1 -- 172.21.15.12:0/1805194177 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf41a20c0 con 0x7f0bf4104bf0 2024-03-17T23:55:19.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.230+0000 7f0bf9200640 1 -- 172.21.15.12:0/1805194177 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bf410acc0 con 0x7f0bf4104bf0 2024-03-17T23:55:19.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.232+0000 7f0beb400640 1 -- 172.21.15.12:0/1805194177 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0bdc0046c0 con 0x7f0bf4104bf0 2024-03-17T23:55:19.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.233+0000 7f0beb400640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0bd40a2800 0x7f0bd40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:19.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.234+0000 7f0beb400640 1 -- 172.21.15.12:0/1805194177 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f0bdc0cf0e0 con 0x7f0bf4104bf0 2024-03-17T23:55:19.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.234+0000 7f0bf1e00640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0bd40a2800 0x7f0bd40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:19.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.234+0000 7f0bf1e00640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0bd40a2800 0x7f0bd40a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0bf419ab60 tx=0x7f0be4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:19.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.241+0000 7f0beb400640 1 -- 172.21.15.12:0/1805194177 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0bdc09bee0 con 0x7f0bf4104bf0 2024-03-17T23:55:19.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:18 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/95410430' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:19.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:18 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/205850879' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:55:19.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:18 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/95410430' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:19.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:18 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/205850879' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:55:19.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:18 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/95410430' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:19.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:18 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/205850879' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:55:19.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.444+0000 7f0bf9200640 1 -- 172.21.15.12:0/1805194177 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0bf4062480 con 0x7f0bf4104bf0 2024-03-17T23:55:19.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.445+0000 7f0beb400640 1 -- 172.21.15.12:0/1805194177 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f0bdc09b7e0 con 0x7f0bf4104bf0 2024-03-17T23:55:19.446 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-17T23:55:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.448+0000 7f0bf9200640 1 -- 172.21.15.12:0/1805194177 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0bd40a2800 msgr2=0x7f0bd40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.448+0000 7f0bf9200640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0bd40a2800 0x7f0bd40a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0bf419ab60 tx=0x7f0be4009260 comp rx=0 tx=0).stop 2024-03-17T23:55:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.448+0000 7f0bf9200640 1 -- 172.21.15.12:0/1805194177 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0bf4104bf0 msgr2=0x7f0bf419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.448+0000 7f0bf9200640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0bf4104bf0 0x7f0bf419a1d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc00e9f0 tx=0x7f0bdc00edf0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.448+0000 7f0bf9200640 1 -- 172.21.15.12:0/1805194177 shutdown_connections 2024-03-17T23:55:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.448+0000 7f0bf9200640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0bd40a2800 0x7f0bd40a4cc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.448+0000 7f0bf9200640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bf41055b0 0x7f0bf41a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.448+0000 7f0bf9200640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0bf4104bf0 0x7f0bf419a1d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.448+0000 7f0bf9200640 1 --2- 172.21.15.12:0/1805194177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bf4103a00 0x7f0bf4199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.448+0000 7f0bf9200640 1 -- 172.21.15.12:0/1805194177 >> 172.21.15.12:0/1805194177 conn(0x7f0bf40fef90 msgr2=0x7f0bf4107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.448+0000 7f0bf9200640 1 -- 172.21.15.12:0/1805194177 shutdown_connections 2024-03-17T23:55:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.448+0000 7f0bf9200640 1 -- 172.21.15.12:0/1805194177 wait complete. 2024-03-17T23:55:19.469 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:55:19.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.695+0000 7f9f0d000640 1 -- 172.21.15.12:0/237174508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f08102140 msgr2=0x7f9f08104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:19.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.695+0000 7f9f0d000640 1 --2- 172.21.15.12:0/237174508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f08102140 0x7f9f08104530 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8009ca0 tx=0x7f9ef802c0f0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.695+0000 7f9f0d000640 1 -- 172.21.15.12:0/237174508 shutdown_connections 2024-03-17T23:55:19.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.695+0000 7f9f0d000640 1 --2- 172.21.15.12:0/237174508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f081073a0 0x7f9f081097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.695+0000 7f9f0d000640 1 --2- 172.21.15.12:0/237174508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f08104a70 0x7f9f08106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.695+0000 7f9f0d000640 1 --2- 172.21.15.12:0/237174508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f08102140 0x7f9f08104530 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.695+0000 7f9f0d000640 1 -- 172.21.15.12:0/237174508 >> 172.21.15.12:0/237174508 conn(0x7f9f080fbb50 msgr2=0x7f9f080fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:19.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.695+0000 7f9f0d000640 1 -- 172.21.15.12:0/237174508 shutdown_connections 2024-03-17T23:55:19.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.695+0000 7f9f0d000640 1 -- 172.21.15.12:0/237174508 wait complete. 2024-03-17T23:55:19.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.696+0000 7f9f0d000640 1 Processor -- start 2024-03-17T23:55:19.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.697+0000 7f9f0d000640 1 -- start start 2024-03-17T23:55:19.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.697+0000 7f9f0d000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f08102140 0x7f9f08199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:19.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.697+0000 7f9f0d000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f08104a70 0x7f9f0819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.697+0000 7f9f0d000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f081073a0 0x7f9f081a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.697+0000 7f9f0d000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0810d340 con 0x7f9f08104a70 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.697+0000 7f9f0d000640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0810d1c0 con 0x7f9f08102140 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.697+0000 7f9f0d000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0810d4c0 con 0x7f9f081073a0 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.697+0000 7f9f07400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f08102140 0x7f9f08199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.697+0000 7f9f07400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f08102140 0x7f9f08199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:53050/0 (socket says 172.21.15.12:53050) 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.697+0000 7f9efea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f08104a70 0x7f9f0819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.697+0000 7f9f07400640 1 -- 172.21.15.12:0/2099670504 learned_addr learned my addr 172.21.15.12:0/2099670504 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.697+0000 7f9f07e00640 1 --2- 172.21.15.12:0/2099670504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f081073a0 0x7f9f081a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.698+0000 7f9f07400640 1 -- 172.21.15.12:0/2099670504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f081073a0 msgr2=0x7f9f081a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.698+0000 7f9f07400640 1 --2- 172.21.15.12:0/2099670504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f081073a0 0x7f9f081a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.698+0000 7f9f07400640 1 -- 172.21.15.12:0/2099670504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f08104a70 msgr2=0x7f9f0819a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.698+0000 7f9f07400640 1 --2- 172.21.15.12:0/2099670504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f08104a70 0x7f9f0819a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.698+0000 7f9f07400640 1 -- 172.21.15.12:0/2099670504 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f081a17f0 con 0x7f9f08102140 2024-03-17T23:55:19.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.698+0000 7f9f07400640 1 --2- 172.21.15.12:0/2099670504 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f08102140 0x7f9f08199bd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80325f0 tx=0x7f9ef8032ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:19.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.698+0000 7f9f04c00640 1 -- 172.21.15.12:0/2099670504 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8041020 con 0x7f9f08102140 2024-03-17T23:55:19.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.698+0000 7f9f04c00640 1 -- 172.21.15.12:0/2099670504 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ef8034cd0 con 0x7f9f08102140 2024-03-17T23:55:19.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.698+0000 7f9f04c00640 1 -- 172.21.15.12:0/2099670504 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8039040 con 0x7f9f08102140 2024-03-17T23:55:19.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.698+0000 7f9f0d000640 1 -- 172.21.15.12:0/2099670504 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f081a1a80 con 0x7f9f08102140 2024-03-17T23:55:19.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.699+0000 7f9f0d000640 1 -- 172.21.15.12:0/2099670504 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f081a1ef0 con 0x7f9f08102140 2024-03-17T23:55:19.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.700+0000 7f9f0d000640 1 -- 172.21.15.12:0/2099670504 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f08108ef0 con 0x7f9f08102140 2024-03-17T23:55:19.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.701+0000 7f9f04c00640 1 -- 172.21.15.12:0/2099670504 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9ef8044020 con 0x7f9f08102140 2024-03-17T23:55:19.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.702+0000 7f9f04c00640 1 --2- 172.21.15.12:0/2099670504 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9eec0a2800 0x7f9eec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:19.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.703+0000 7f9f04c00640 1 -- 172.21.15.12:0/2099670504 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f9ef80f0bb0 con 0x7f9f08102140 2024-03-17T23:55:19.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.703+0000 7f9efea00640 1 --2- 172.21.15.12:0/2099670504 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9eec0a2800 0x7f9eec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:19.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.703+0000 7f9efea00640 1 --2- 172.21.15.12:0/2099670504 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9eec0a2800 0x7f9eec0a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9f0819b0e0 tx=0x7f9ef0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:19.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.707+0000 7f9f04c00640 1 -- 172.21.15.12:0/2099670504 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9ef804b020 con 0x7f9f08102140 2024-03-17T23:55:19.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.914+0000 7f9f0d000640 1 -- 172.21.15.12:0/2099670504 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9f080623e0 con 0x7f9f08102140 2024-03-17T23:55:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.914+0000 7f9f04c00640 1 -- 172.21.15.12:0/2099670504 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f9ef80b7e20 con 0x7f9f08102140 2024-03-17T23:55:19.915 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-17T23:55:19.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.917+0000 7f9f0d000640 1 -- 172.21.15.12:0/2099670504 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9eec0a2800 msgr2=0x7f9eec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:19.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.917+0000 7f9f0d000640 1 --2- 172.21.15.12:0/2099670504 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9eec0a2800 0x7f9eec0a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9f0819b0e0 tx=0x7f9ef0005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:19.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.917+0000 7f9f0d000640 1 -- 172.21.15.12:0/2099670504 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f08102140 msgr2=0x7f9f08199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:19.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.917+0000 7f9f0d000640 1 --2- 172.21.15.12:0/2099670504 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f08102140 0x7f9f08199bd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80325f0 tx=0x7f9ef8032ec0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.917+0000 7f9f0d000640 1 -- 172.21.15.12:0/2099670504 shutdown_connections 2024-03-17T23:55:19.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.917+0000 7f9f0d000640 1 --2- 172.21.15.12:0/2099670504 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9eec0a2800 0x7f9eec0a4cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.917+0000 7f9f0d000640 1 --2- 172.21.15.12:0/2099670504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f081073a0 0x7f9f081a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.917+0000 7f9f0d000640 1 --2- 172.21.15.12:0/2099670504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f08104a70 0x7f9f0819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.917+0000 7f9f0d000640 1 --2- 172.21.15.12:0/2099670504 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f08102140 0x7f9f08199bd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:19.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.917+0000 7f9f0d000640 1 -- 172.21.15.12:0/2099670504 >> 172.21.15.12:0/2099670504 conn(0x7f9f080fbb50 msgr2=0x7f9f081017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:19.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.917+0000 7f9f0d000640 1 -- 172.21.15.12:0/2099670504 shutdown_connections 2024-03-17T23:55:19.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:19.917+0000 7f9f0d000640 1 -- 172.21.15.12:0/2099670504 wait complete. 2024-03-17T23:55:19.938 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:55:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.157+0000 7f1927a00640 1 -- 172.21.15.12:0/337879754 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19201033a0 msgr2=0x7f1920109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.157+0000 7f1927a00640 1 --2- 172.21.15.12:0/337879754 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19201033a0 0x7f1920109c30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1918009ca0 tx=0x7f191802c100 comp rx=0 tx=0).stop 2024-03-17T23:55:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.157+0000 7f1927a00640 1 -- 172.21.15.12:0/337879754 shutdown_connections 2024-03-17T23:55:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.157+0000 7f1927a00640 1 --2- 172.21.15.12:0/337879754 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19201033a0 0x7f1920109c30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.157+0000 7f1927a00640 1 --2- 172.21.15.12:0/337879754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19201029e0 0x7f1920102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.157+0000 7f1927a00640 1 --2- 172.21.15.12:0/337879754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19201017f0 0x7f1920101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.157+0000 7f1927a00640 1 -- 172.21.15.12:0/337879754 >> 172.21.15.12:0/337879754 conn(0x7f19200fcd80 msgr2=0x7f19200ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.157+0000 7f1927a00640 1 -- 172.21.15.12:0/337879754 shutdown_connections 2024-03-17T23:55:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.157+0000 7f1927a00640 1 -- 172.21.15.12:0/337879754 wait complete. 2024-03-17T23:55:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.158+0000 7f1927a00640 1 Processor -- start 2024-03-17T23:55:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.158+0000 7f1927a00640 1 -- start start 2024-03-17T23:55:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.159+0000 7f1927a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19201017f0 0x7f1920199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.159+0000 7f1927a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19201029e0 0x7f192019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.159+0000 7f1927a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19201033a0 0x7f19201a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.159+0000 7f1927a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f192010d450 con 0x7f19201033a0 2024-03-17T23:55:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.159+0000 7f1927a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f192010d2d0 con 0x7f19201017f0 2024-03-17T23:55:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.159+0000 7f1927a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f192010d5d0 con 0x7f19201029e0 2024-03-17T23:55:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.159+0000 7f1925000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19201017f0 0x7f1920199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.159+0000 7f1925000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19201017f0 0x7f1920199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:53058/0 (socket says 172.21.15.12:53058) 2024-03-17T23:55:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.159+0000 7f1925000640 1 -- 172.21.15.12:0/1350215838 learned_addr learned my addr 172.21.15.12:0/1350215838 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.159+0000 7f1925a00640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19201033a0 0x7f19201a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.159+0000 7f191fe00640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19201029e0 0x7f192019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.160+0000 7f1925000640 1 -- 172.21.15.12:0/1350215838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19201029e0 msgr2=0x7f192019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.160+0000 7f1925000640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19201029e0 0x7f192019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.160+0000 7f1925000640 1 -- 172.21.15.12:0/1350215838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19201033a0 msgr2=0x7f19201a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.160+0000 7f1925000640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19201033a0 0x7f19201a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.160+0000 7f1925000640 1 -- 172.21.15.12:0/1350215838 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19201a1850 con 0x7f19201017f0 2024-03-17T23:55:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.160+0000 7f1925000640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19201017f0 0x7f1920199c40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f191400d770 tx=0x7f191400db70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.160+0000 7f191d600640 1 -- 172.21.15.12:0/1350215838 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1914017070 con 0x7f19201017f0 2024-03-17T23:55:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.160+0000 7f191d600640 1 -- 172.21.15.12:0/1350215838 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1914009d00 con 0x7f19201017f0 2024-03-17T23:55:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.160+0000 7f1927a00640 1 -- 172.21.15.12:0/1350215838 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19201a1b40 con 0x7f19201017f0 2024-03-17T23:55:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.160+0000 7f191d600640 1 -- 172.21.15.12:0/1350215838 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1914003d30 con 0x7f19201017f0 2024-03-17T23:55:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.160+0000 7f1927a00640 1 -- 172.21.15.12:0/1350215838 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19201a20e0 con 0x7f19201017f0 2024-03-17T23:55:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.161+0000 7f1927a00640 1 -- 172.21.15.12:0/1350215838 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19201a22b0 con 0x7f19201017f0 2024-03-17T23:55:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.164+0000 7f191d600640 1 -- 172.21.15.12:0/1350215838 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1914009820 con 0x7f19201017f0 2024-03-17T23:55:20.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.165+0000 7f191d600640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19080a2880 0x7f19080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:20.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.165+0000 7f191d600640 1 -- 172.21.15.12:0/1350215838 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f19140ce5e0 con 0x7f19201017f0 2024-03-17T23:55:20.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.165+0000 7f191fe00640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19080a2880 0x7f19080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:20.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.166+0000 7f191fe00640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19080a2880 0x7f19080a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f190c00abe0 tx=0x7f190c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:20.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.169+0000 7f191d600640 1 -- 172.21.15.12:0/1350215838 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f191409b3e0 con 0x7f19201017f0 2024-03-17T23:55:20.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:19 smithi012 ceph-mon[31541]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-17T23:55:20.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:19 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4274803250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:55:20.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:19 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1805194177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:55:20.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:19 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2099670504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:55:20.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:19 smithi079 ceph-mon[36241]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-17T23:55:20.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:19 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/4274803250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:55:20.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:19 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1805194177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:55:20.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:19 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2099670504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:55:20.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:19 smithi191 ceph-mon[36708]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-17T23:55:20.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:19 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/4274803250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:55:20.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:19 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1805194177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:55:20.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:19 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2099670504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:55:20.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.397+0000 7f1927a00640 1 -- 172.21.15.12:0/1350215838 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f19201a2520 con 0x7f19201017f0 2024-03-17T23:55:20.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.397+0000 7f191d600640 1 -- 172.21.15.12:0/1350215838 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+590 (secure 0 0 0) 0x7f191409ace0 con 0x7f19201017f0 2024-03-17T23:55:20.398 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/2526360059 2024-03-18T23:46:53.955890+0000 2024-03-17T23:55:20.398 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/2336882461 2024-03-18T23:46:53.955890+0000 2024-03-17T23:55:20.398 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/1563339918 2024-03-18T23:46:19.706199+0000 2024-03-17T23:55:20.398 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1976882257 2024-03-18T23:46:53.955890+0000 2024-03-17T23:55:20.398 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/501542164 2024-03-18T23:46:53.955890+0000 2024-03-17T23:55:20.399 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/654354506 2024-03-18T23:46:19.706199+0000 2024-03-17T23:55:20.399 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/1563339918 2024-03-18T23:46:19.706199+0000 2024-03-17T23:55:20.399 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1974593432 2024-03-18T23:46:19.706199+0000 2024-03-17T23:55:20.399 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/2526360059 2024-03-18T23:46:53.955890+0000 2024-03-17T23:55:20.399 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1690054132 2024-03-18T23:46:19.706199+0000 2024-03-17T23:55:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:listed 10 entries 2024-03-17T23:55:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.400+0000 7f1927a00640 1 -- 172.21.15.12:0/1350215838 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19080a2880 msgr2=0x7f19080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.400+0000 7f1927a00640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19080a2880 0x7f19080a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f190c00abe0 tx=0x7f190c00a2d0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.400+0000 7f1927a00640 1 -- 172.21.15.12:0/1350215838 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19201017f0 msgr2=0x7f1920199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.400+0000 7f1927a00640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19201017f0 0x7f1920199c40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f191400d770 tx=0x7f191400db70 comp rx=0 tx=0).stop 2024-03-17T23:55:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.400+0000 7f1927a00640 1 -- 172.21.15.12:0/1350215838 shutdown_connections 2024-03-17T23:55:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.400+0000 7f1927a00640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19080a2880 0x7f19080a4d40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.400+0000 7f1927a00640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19201033a0 0x7f19201a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.400+0000 7f1927a00640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19201029e0 0x7f192019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.400+0000 7f1927a00640 1 --2- 172.21.15.12:0/1350215838 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19201017f0 0x7f1920199c40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.400+0000 7f1927a00640 1 -- 172.21.15.12:0/1350215838 >> 172.21.15.12:0/1350215838 conn(0x7f19200fcd80 msgr2=0x7f1920073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:20.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.400+0000 7f1927a00640 1 -- 172.21.15.12:0/1350215838 shutdown_connections 2024-03-17T23:55:20.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.400+0000 7f1927a00640 1 -- 172.21.15.12:0/1350215838 wait complete. 2024-03-17T23:55:20.420 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi012.front.sepia.ceph.com... 2024-03-17T23:55:20.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:55:20.420 DEBUG:teuthology.orchestra.run.smithi012:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:55:20.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi012.stderr:ceph-fuse[72602]: fuse finished with error 0 and tester_r 0 2024-03-17T23:55:20.519 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:55:20.589 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-17T23:55:20.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:55:20.589 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-17T23:55:20.658 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:55:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.887+0000 7f2705200640 1 -- 172.21.15.12:0/3996879946 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27001053c0 msgr2=0x7f270010bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.887+0000 7f2705200640 1 --2- 172.21.15.12:0/3996879946 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27001053c0 0x7f270010bc50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009900 tx=0x7f26f802bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.887+0000 7f2705200640 1 -- 172.21.15.12:0/3996879946 shutdown_connections 2024-03-17T23:55:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.887+0000 7f2705200640 1 --2- 172.21.15.12:0/3996879946 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27001053c0 0x7f270010bc50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.887+0000 7f2705200640 1 --2- 172.21.15.12:0/3996879946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2700104a00 0x7f2700104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.887+0000 7f2705200640 1 --2- 172.21.15.12:0/3996879946 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2700103810 0x7f2700103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.887+0000 7f2705200640 1 -- 172.21.15.12:0/3996879946 >> 172.21.15.12:0/3996879946 conn(0x7f27000fed80 msgr2=0x7f27001011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.887+0000 7f2705200640 1 -- 172.21.15.12:0/3996879946 shutdown_connections 2024-03-17T23:55:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.887+0000 7f2705200640 1 -- 172.21.15.12:0/3996879946 wait complete. 2024-03-17T23:55:20.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.888+0000 7f2705200640 1 Processor -- start 2024-03-17T23:55:20.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.888+0000 7f2705200640 1 -- start start 2024-03-17T23:55:20.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.888+0000 7f2705200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2700103810 0x7f2700199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:20.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f2705200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2700104a00 0x7f2700199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:20.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f2705200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27001053c0 0x7f27001a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:20.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f2705200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f270010f490 con 0x7f2700103810 2024-03-17T23:55:20.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f2705200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f270010f310 con 0x7f2700104a00 2024-03-17T23:55:20.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f2705200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f270010f610 con 0x7f27001053c0 2024-03-17T23:55:20.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f26ff400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2700103810 0x7f2700199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:20.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f26f6a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2700104a00 0x7f2700199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:20.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f26ff400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2700103810 0x7f2700199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32796/0 (socket says 172.21.15.12:32796) 2024-03-17T23:55:20.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f26ff400640 1 -- 172.21.15.12:0/3943600737 learned_addr learned my addr 172.21.15.12:0/3943600737 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:20.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f26ffe00640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27001053c0 0x7f27001a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:20.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f26f6a00640 1 -- 172.21.15.12:0/3943600737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27001053c0 msgr2=0x7f27001a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:20.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f26f6a00640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27001053c0 0x7f27001a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f26f6a00640 1 -- 172.21.15.12:0/3943600737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2700103810 msgr2=0x7f2700199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:20.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f26f6a00640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2700103810 0x7f2700199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:20.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f26f6a00640 1 -- 172.21.15.12:0/3943600737 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27001a16a0 con 0x7f2700104a00 2024-03-17T23:55:20.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.889+0000 7f26ff400640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2700103810 0x7f2700199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:55:20.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.890+0000 7f26f6a00640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2700104a00 0x7f2700199fc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f26e800eb00 tx=0x7f26e800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:20.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.890+0000 7f26fcc00640 1 -- 172.21.15.12:0/3943600737 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e8010430 con 0x7f2700104a00 2024-03-17T23:55:20.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.890+0000 7f26fcc00640 1 -- 172.21.15.12:0/3943600737 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26e8010ab0 con 0x7f2700104a00 2024-03-17T23:55:20.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.890+0000 7f26fcc00640 1 -- 172.21.15.12:0/3943600737 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e8017bb0 con 0x7f2700104a00 2024-03-17T23:55:20.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.890+0000 7f2705200640 1 -- 172.21.15.12:0/3943600737 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27001a1990 con 0x7f2700104a00 2024-03-17T23:55:20.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.890+0000 7f2705200640 1 -- 172.21.15.12:0/3943600737 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27001a1e90 con 0x7f2700104a00 2024-03-17T23:55:20.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.891+0000 7f2705200640 1 -- 172.21.15.12:0/3943600737 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f270010aad0 con 0x7f2700104a00 2024-03-17T23:55:20.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.893+0000 7f26fcc00640 1 -- 172.21.15.12:0/3943600737 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f26e8010c60 con 0x7f2700104a00 2024-03-17T23:55:20.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.894+0000 7f26fcc00640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f26e40a2880 0x7f26e40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:20.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.895+0000 7f26fcc00640 1 -- 172.21.15.12:0/3943600737 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f26e80cf690 con 0x7f2700104a00 2024-03-17T23:55:20.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.895+0000 7f26ff400640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f26e40a2880 0x7f26e40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:20.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.895+0000 7f26ff400640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f26e40a2880 0x7f26e40a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f26f00058c0 tx=0x7f26f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:20.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:20.896+0000 7f26fcc00640 1 -- 172.21.15.12:0/3943600737 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f26e809c490 con 0x7f2700104a00 2024-03-17T23:55:21.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.103+0000 7f2705200640 1 -- 172.21.15.12:0/3943600737 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f270002d400 con 0x7f2700104a00 2024-03-17T23:55:21.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.105+0000 7f26fcc00640 1 -- 172.21.15.12:0/3943600737 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24044 (secure 0 0 0) 0x7f26e809bd90 con 0x7f2700104a00 2024-03-17T23:55:21.106 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:21.106 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":90,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:55:02.337441+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:55:21.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.108+0000 7f2705200640 1 -- 172.21.15.12:0/3943600737 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f26e40a2880 msgr2=0x7f26e40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:21.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.108+0000 7f2705200640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f26e40a2880 0x7f26e40a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f26f00058c0 tx=0x7f26f0005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:21.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.108+0000 7f2705200640 1 -- 172.21.15.12:0/3943600737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2700104a00 msgr2=0x7f2700199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:21.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.108+0000 7f2705200640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2700104a00 0x7f2700199fc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f26e800eb00 tx=0x7f26e800c2d0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.108+0000 7f2705200640 1 -- 172.21.15.12:0/3943600737 shutdown_connections 2024-03-17T23:55:21.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.108+0000 7f2705200640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f26e40a2880 0x7f26e40a4d40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.108+0000 7f2705200640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27001053c0 0x7f27001a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.108+0000 7f2705200640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2700104a00 0x7f2700199fc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.108+0000 7f2705200640 1 --2- 172.21.15.12:0/3943600737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2700103810 0x7f2700199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.108+0000 7f2705200640 1 -- 172.21.15.12:0/3943600737 >> 172.21.15.12:0/3943600737 conn(0x7f27000fed80 msgr2=0x7f2700107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:21.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.109+0000 7f2705200640 1 -- 172.21.15.12:0/3943600737 shutdown_connections 2024-03-17T23:55:21.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.109+0000 7f2705200640 1 -- 172.21.15.12:0/3943600737 wait complete. 2024-03-17T23:55:21.128 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:55:21.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:20 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1350215838' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-17T23:55:21.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:20 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1350215838' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-17T23:55:21.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:20 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1350215838' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-17T23:55:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.347+0000 7fccc5400640 1 -- 172.21.15.12:0/3342249167 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fccc0100820 msgr2=0x7fccc0100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.347+0000 7fccc5400640 1 --2- 172.21.15.12:0/3342249167 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fccc0100820 0x7fccc0100c80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcca8009900 tx=0x7fcca802bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.347+0000 7fccc5400640 1 -- 172.21.15.12:0/3342249167 shutdown_connections 2024-03-17T23:55:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.347+0000 7fccc5400640 1 --2- 172.21.15.12:0/3342249167 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fccc01011c0 0x7fccc0101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.347+0000 7fccc5400640 1 --2- 172.21.15.12:0/3342249167 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fccc0100820 0x7fccc0100c80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.347+0000 7fccc5400640 1 --2- 172.21.15.12:0/3342249167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccc00ffec0 0x7fccc01002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.347+0000 7fccc5400640 1 -- 172.21.15.12:0/3342249167 >> 172.21.15.12:0/3342249167 conn(0x7fccc00fba60 msgr2=0x7fccc00fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.347+0000 7fccc5400640 1 -- 172.21.15.12:0/3342249167 shutdown_connections 2024-03-17T23:55:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.347+0000 7fccc5400640 1 -- 172.21.15.12:0/3342249167 wait complete. 2024-03-17T23:55:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.348+0000 7fccc5400640 1 Processor -- start 2024-03-17T23:55:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.348+0000 7fccc5400640 1 -- start start 2024-03-17T23:55:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccc5400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fccc00ffec0 0x7fccc0199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccc5400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fccc0100820 0x7fccc019a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccc5400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccc01011c0 0x7fccc01a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccc5400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fccc010f460 con 0x7fccc01011c0 2024-03-17T23:55:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccc5400640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fccc010f2e0 con 0x7fccc0100820 2024-03-17T23:55:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccc5400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fccc010f5e0 con 0x7fccc00ffec0 2024-03-17T23:55:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccbf400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fccc0100820 0x7fccc019a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccbf400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fccc0100820 0x7fccc019a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:53092/0 (socket says 172.21.15.12:53092) 2024-03-17T23:55:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccbf400640 1 -- 172.21.15.12:0/2953773288 learned_addr learned my addr 172.21.15.12:0/2953773288 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccc4a00640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccc01011c0 0x7fccc01a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccbfe00640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fccc00ffec0 0x7fccc0199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:21.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccbf400640 1 -- 172.21.15.12:0/2953773288 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fccc00ffec0 msgr2=0x7fccc0199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:21.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccbf400640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fccc00ffec0 0x7fccc0199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccbf400640 1 -- 172.21.15.12:0/2953773288 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccc01011c0 msgr2=0x7fccc01a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:21.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccbf400640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccc01011c0 0x7fccc01a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.349+0000 7fccbf400640 1 -- 172.21.15.12:0/2953773288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc01a16f0 con 0x7fccc0100820 2024-03-17T23:55:21.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.350+0000 7fccbf400640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fccc0100820 0x7fccc019a010 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcca8009230 tx=0x7fcca802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:21.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.350+0000 7fccbcc00640 1 -- 172.21.15.12:0/2953773288 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca8041070 con 0x7fccc0100820 2024-03-17T23:55:21.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.350+0000 7fccbcc00640 1 -- 172.21.15.12:0/2953773288 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcca8004450 con 0x7fccc0100820 2024-03-17T23:55:21.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.350+0000 7fccbcc00640 1 -- 172.21.15.12:0/2953773288 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca8003e20 con 0x7fccc0100820 2024-03-17T23:55:21.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.350+0000 7fccc5400640 1 -- 172.21.15.12:0/2953773288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc01a1980 con 0x7fccc0100820 2024-03-17T23:55:21.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.350+0000 7fccc5400640 1 -- 172.21.15.12:0/2953773288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc01a1e20 con 0x7fccc0100820 2024-03-17T23:55:21.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.351+0000 7fccc5400640 1 -- 172.21.15.12:0/2953773288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccc010ac10 con 0x7fccc0100820 2024-03-17T23:55:21.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.353+0000 7fccbcc00640 1 -- 172.21.15.12:0/2953773288 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcca8044070 con 0x7fccc0100820 2024-03-17T23:55:21.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.354+0000 7fccbcc00640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcca40a2880 0x7fcca40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:21.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.355+0000 7fccbcc00640 1 -- 172.21.15.12:0/2953773288 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fcca80f0800 con 0x7fccc0100820 2024-03-17T23:55:21.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.355+0000 7fccbfe00640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcca40a2880 0x7fcca40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:21.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.356+0000 7fccbfe00640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcca40a2880 0x7fcca40a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fccb0005fd0 tx=0x7fccb0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.362+0000 7fccbcc00640 1 -- 172.21.15.12:0/2953773288 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcca80bcfb0 con 0x7fccc0100820 2024-03-17T23:55:21.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.565+0000 7fccc5400640 1 -- 172.21.15.12:0/2953773288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fccc0062480 con 0x7fccc0100820 2024-03-17T23:55:21.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.568+0000 7fccbcc00640 1 -- 172.21.15.12:0/2953773288 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24044 (secure 0 0 0) 0x7fcca809be20 con 0x7fccc0100820 2024-03-17T23:55:21.569 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:21.569 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":90,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:55:02.337441+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:55:21.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.570+0000 7fccc5400640 1 -- 172.21.15.12:0/2953773288 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcca40a2880 msgr2=0x7fcca40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:21.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.570+0000 7fccc5400640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcca40a2880 0x7fcca40a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fccb0005fd0 tx=0x7fccb0009500 comp rx=0 tx=0).stop 2024-03-17T23:55:21.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.570+0000 7fccc5400640 1 -- 172.21.15.12:0/2953773288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fccc0100820 msgr2=0x7fccc019a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:21.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.570+0000 7fccc5400640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fccc0100820 0x7fccc019a010 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcca8009230 tx=0x7fcca802c650 comp rx=0 tx=0).stop 2024-03-17T23:55:21.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.570+0000 7fccc5400640 1 -- 172.21.15.12:0/2953773288 shutdown_connections 2024-03-17T23:55:21.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.570+0000 7fccc5400640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcca40a2880 0x7fcca40a4d40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.570+0000 7fccc5400640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccc01011c0 0x7fccc01a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.570+0000 7fccc5400640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fccc0100820 0x7fccc019a010 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.570+0000 7fccc5400640 1 --2- 172.21.15.12:0/2953773288 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fccc00ffec0 0x7fccc0199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.570+0000 7fccc5400640 1 -- 172.21.15.12:0/2953773288 >> 172.21.15.12:0/2953773288 conn(0x7fccc00fba60 msgr2=0x7fccc00fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:21.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.570+0000 7fccc5400640 1 -- 172.21.15.12:0/2953773288 shutdown_connections 2024-03-17T23:55:21.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.570+0000 7fccc5400640 1 -- 172.21.15.12:0/2953773288 wait complete. 2024-03-17T23:55:21.591 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-17T23:55:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.808+0000 7fd421c00640 1 -- 172.21.15.12:0/2468508401 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1033a0 msgr2=0x7fd41c109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.808+0000 7fd421c00640 1 --2- 172.21.15.12:0/2468508401 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1033a0 0x7fd41c109c30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd40c009900 tx=0x7fd40c02bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.808+0000 7fd421c00640 1 -- 172.21.15.12:0/2468508401 shutdown_connections 2024-03-17T23:55:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.808+0000 7fd421c00640 1 --2- 172.21.15.12:0/2468508401 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1033a0 0x7fd41c109c30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.808+0000 7fd421c00640 1 --2- 172.21.15.12:0/2468508401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd41c1029e0 0x7fd41c102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.808+0000 7fd421c00640 1 --2- 172.21.15.12:0/2468508401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd41c1017f0 0x7fd41c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.808+0000 7fd421c00640 1 -- 172.21.15.12:0/2468508401 >> 172.21.15.12:0/2468508401 conn(0x7fd41c0fcd80 msgr2=0x7fd41c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.808+0000 7fd421c00640 1 -- 172.21.15.12:0/2468508401 shutdown_connections 2024-03-17T23:55:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.808+0000 7fd421c00640 1 -- 172.21.15.12:0/2468508401 wait complete. 2024-03-17T23:55:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.809+0000 7fd421c00640 1 Processor -- start 2024-03-17T23:55:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.809+0000 7fd421c00640 1 -- start start 2024-03-17T23:55:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd421c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1017f0 0x7fd41c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd421c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd41c1029e0 0x7fd41c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd421c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd41c1033a0 0x7fd41c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd421c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c10d4a0 con 0x7fd41c1033a0 2024-03-17T23:55:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd421c00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c10d320 con 0x7fd41c1017f0 2024-03-17T23:55:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd421c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c10d620 con 0x7fd41c1029e0 2024-03-17T23:55:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd41aa00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1017f0 0x7fd41c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd41aa00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1017f0 0x7fd41c197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:53102/0 (socket says 172.21.15.12:53102) 2024-03-17T23:55:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd41aa00640 1 -- 172.21.15.12:0/1313159240 learned_addr learned my addr 172.21.15.12:0/1313159240 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd41b400640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd41c1033a0 0x7fd41c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd41a000640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd41c1029e0 0x7fd41c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd41aa00640 1 -- 172.21.15.12:0/1313159240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd41c1029e0 msgr2=0x7fd41c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd41aa00640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd41c1029e0 0x7fd41c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd41aa00640 1 -- 172.21.15.12:0/1313159240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd41c1033a0 msgr2=0x7fd41c19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:21.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.810+0000 7fd41aa00640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd41c1033a0 0x7fd41c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:21.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.811+0000 7fd41aa00640 1 -- 172.21.15.12:0/1313159240 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41c19f6a0 con 0x7fd41c1017f0 2024-03-17T23:55:21.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.811+0000 7fd41a000640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd41c1029e0 0x7fd41c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:55:21.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.811+0000 7fd41aa00640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1017f0 0x7fd41c197a80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd41400ea80 tx=0x7fd41400ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:21.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.811+0000 7fd413400640 1 -- 172.21.15.12:0/1313159240 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41400ce40 con 0x7fd41c1017f0 2024-03-17T23:55:21.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.811+0000 7fd413400640 1 -- 172.21.15.12:0/1313159240 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd414011950 con 0x7fd41c1017f0 2024-03-17T23:55:21.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.811+0000 7fd413400640 1 -- 172.21.15.12:0/1313159240 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41401ab80 con 0x7fd41c1017f0 2024-03-17T23:55:21.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.811+0000 7fd421c00640 1 -- 172.21.15.12:0/1313159240 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd41c19f990 con 0x7fd41c1017f0 2024-03-17T23:55:21.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.811+0000 7fd421c00640 1 -- 172.21.15.12:0/1313159240 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd41c19ff30 con 0x7fd41c1017f0 2024-03-17T23:55:21.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.812+0000 7fd421c00640 1 -- 172.21.15.12:0/1313159240 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd41c068a20 con 0x7fd41c1017f0 2024-03-17T23:55:21.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.814+0000 7fd413400640 1 -- 172.21.15.12:0/1313159240 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd414011b00 con 0x7fd41c1017f0 2024-03-17T23:55:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.815+0000 7fd413400640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd4000a2880 0x7fd4000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.816+0000 7fd413400640 1 -- 172.21.15.12:0/1313159240 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd4140cf2e0 con 0x7fd41c1017f0 2024-03-17T23:55:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.816+0000 7fd41a000640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd4000a2880 0x7fd4000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:21.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.817+0000 7fd41a000640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd4000a2880 0x7fd4000a4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd41c198f90 tx=0x7fd408005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:21.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:21.824+0000 7fd413400640 1 -- 172.21.15.12:0/1313159240 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd414016030 con 0x7fd41c1017f0 2024-03-17T23:55:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.069+0000 7fd421c00640 1 -- 172.21.15.12:0/1313159240 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd41c108b10 con 0x7fd41c1017f0 2024-03-17T23:55:22.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:21 smithi079 ceph-mon[36241]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-17T23:55:22.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:21 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3943600737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:22.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:21 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2953773288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:22.327 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:21 smithi012 ceph-mon[31541]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-17T23:55:22.328 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:21 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3943600737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:22.328 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:21 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2953773288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:22.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:21 smithi191 ceph-mon[36708]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-17T23:55:22.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:21 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3943600737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:22.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:21 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2953773288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.960+0000 7fd413400640 1 -- 172.21.15.12:0/1313159240 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fd41409c140 con 0x7fd41c1017f0 2024-03-17T23:55:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.964+0000 7fd421c00640 1 -- 172.21.15.12:0/1313159240 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd4000a2880 msgr2=0x7fd4000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.964+0000 7fd421c00640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd4000a2880 0x7fd4000a4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd41c198f90 tx=0x7fd408005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:22.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.964+0000 7fd421c00640 1 -- 172.21.15.12:0/1313159240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1017f0 msgr2=0x7fd41c197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.964+0000 7fd421c00640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1017f0 0x7fd41c197a80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd41400ea80 tx=0x7fd41400ef40 comp rx=0 tx=0).stop 2024-03-17T23:55:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.964+0000 7fd421c00640 1 -- 172.21.15.12:0/1313159240 shutdown_connections 2024-03-17T23:55:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.964+0000 7fd421c00640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd4000a2880 0x7fd4000a4d40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.964+0000 7fd421c00640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd41c1033a0 0x7fd41c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.964+0000 7fd421c00640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd41c1029e0 0x7fd41c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.964+0000 7fd421c00640 1 --2- 172.21.15.12:0/1313159240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1017f0 0x7fd41c197a80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.964+0000 7fd421c00640 1 -- 172.21.15.12:0/1313159240 >> 172.21.15.12:0/1313159240 conn(0x7fd41c0fcd80 msgr2=0x7fd41c105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.965+0000 7fd421c00640 1 -- 172.21.15.12:0/1313159240 shutdown_connections 2024-03-17T23:55:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:22.965+0000 7fd421c00640 1 -- 172.21.15.12:0/1313159240 wait complete. 2024-03-17T23:55:22.987 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-17T23:55:22.987 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-17T23:55:23.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.212+0000 7fa982400640 1 -- 172.21.15.12:0/2220624038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa97c103a00 msgr2=0x7fa97c103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:23.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.212+0000 7fa982400640 1 --2- 172.21.15.12:0/2220624038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa97c103a00 0x7fa97c103e20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009ca0 tx=0x7fa96c02c0f0 comp rx=0 tx=0).stop 2024-03-17T23:55:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.213+0000 7fa982400640 1 -- 172.21.15.12:0/2220624038 shutdown_connections 2024-03-17T23:55:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.213+0000 7fa982400640 1 --2- 172.21.15.12:0/2220624038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa97c1055b0 0x7fa97c10be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.213+0000 7fa982400640 1 --2- 172.21.15.12:0/2220624038 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97c104bf0 0x7fa97c105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.213+0000 7fa982400640 1 --2- 172.21.15.12:0/2220624038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa97c103a00 0x7fa97c103e20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.213+0000 7fa982400640 1 -- 172.21.15.12:0/2220624038 >> 172.21.15.12:0/2220624038 conn(0x7fa97c0fef90 msgr2=0x7fa97c1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.213+0000 7fa982400640 1 -- 172.21.15.12:0/2220624038 shutdown_connections 2024-03-17T23:55:23.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.213+0000 7fa982400640 1 -- 172.21.15.12:0/2220624038 wait complete. 2024-03-17T23:55:23.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.214+0000 7fa982400640 1 Processor -- start 2024-03-17T23:55:23.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.214+0000 7fa982400640 1 -- start start 2024-03-17T23:55:23.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.214+0000 7fa982400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa97c103a00 0x7fa97c199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.214+0000 7fa982400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa97c104bf0 0x7fa97c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.214+0000 7fa982400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97c1055b0 0x7fa97c1a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.214+0000 7fa982400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c10f650 con 0x7fa97c103a00 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.214+0000 7fa982400640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c10f4d0 con 0x7fa97c104bf0 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.214+0000 7fa982400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c10f7d0 con 0x7fa97c1055b0 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.214+0000 7fa97b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa97c103a00 0x7fa97c199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.215+0000 7fa97b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa97c103a00 0x7fa97c199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32854/0 (socket says 172.21.15.12:32854) 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.215+0000 7fa97b400640 1 -- 172.21.15.12:0/800895183 learned_addr learned my addr 172.21.15.12:0/800895183 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.215+0000 7fa972a00640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa97c104bf0 0x7fa97c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.215+0000 7fa97b400640 1 -- 172.21.15.12:0/800895183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97c1055b0 msgr2=0x7fa97c1a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.215+0000 7fa97b400640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97c1055b0 0x7fa97c1a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.215+0000 7fa97b400640 1 -- 172.21.15.12:0/800895183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa97c104bf0 msgr2=0x7fa97c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.215+0000 7fa97b400640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa97c104bf0 0x7fa97c19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:23.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.215+0000 7fa97b400640 1 -- 172.21.15.12:0/800895183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa97c1a1920 con 0x7fa97c103a00 2024-03-17T23:55:23.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.215+0000 7fa972a00640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa97c104bf0 0x7fa97c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:55:23.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.215+0000 7fa97b400640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa97c103a00 0x7fa97c199d10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa96c000f80 tx=0x7fa96c032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:23.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.216+0000 7fa978c00640 1 -- 172.21.15.12:0/800895183 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96c041020 con 0x7fa97c103a00 2024-03-17T23:55:23.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.216+0000 7fa978c00640 1 -- 172.21.15.12:0/800895183 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa96c034cd0 con 0x7fa97c103a00 2024-03-17T23:55:23.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.216+0000 7fa982400640 1 -- 172.21.15.12:0/800895183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa97c1a1bb0 con 0x7fa97c103a00 2024-03-17T23:55:23.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.216+0000 7fa978c00640 1 -- 172.21.15.12:0/800895183 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96c039040 con 0x7fa97c103a00 2024-03-17T23:55:23.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.216+0000 7fa982400640 1 -- 172.21.15.12:0/800895183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa97c1a2070 con 0x7fa97c103a00 2024-03-17T23:55:23.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.217+0000 7fa982400640 1 -- 172.21.15.12:0/800895183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa97c10ada0 con 0x7fa97c103a00 2024-03-17T23:55:23.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.219+0000 7fa978c00640 1 -- 172.21.15.12:0/800895183 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa96c044020 con 0x7fa97c103a00 2024-03-17T23:55:23.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.220+0000 7fa978c00640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa9600a2800 0x7fa9600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:23.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.221+0000 7fa978c00640 1 -- 172.21.15.12:0/800895183 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa96c0f0d00 con 0x7fa97c103a00 2024-03-17T23:55:23.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.221+0000 7fa972a00640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa9600a2800 0x7fa9600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:23.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.222+0000 7fa972a00640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa9600a2800 0x7fa9600a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa9640058c0 tx=0x7fa964005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:23.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.226+0000 7fa978c00640 1 -- 172.21.15.12:0/800895183 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa96c04b020 con 0x7fa97c103a00 2024-03-17T23:55:23.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:22 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1313159240' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-17T23:55:23.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:22 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-17T23:55:23.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:22 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1313159240' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-17T23:55:23.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:22 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-17T23:55:23.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:22 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1313159240' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-17T23:55:23.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:22 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-17T23:55:23.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.480+0000 7fa982400640 1 -- 172.21.15.12:0/800895183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fa97c062480 con 0x7fa97c103a00 2024-03-17T23:55:23.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.959+0000 7fa978c00640 1 -- 172.21.15.12:0/800895183 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fa96c0bdb00 con 0x7fa97c103a00 2024-03-17T23:55:23.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.963+0000 7fa982400640 1 -- 172.21.15.12:0/800895183 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa9600a2800 msgr2=0x7fa9600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:23.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.963+0000 7fa982400640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa9600a2800 0x7fa9600a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa9640058c0 tx=0x7fa964005850 comp rx=0 tx=0).stop 2024-03-17T23:55:23.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.963+0000 7fa982400640 1 -- 172.21.15.12:0/800895183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa97c103a00 msgr2=0x7fa97c199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:23.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.964+0000 7fa982400640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa97c103a00 0x7fa97c199d10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa96c000f80 tx=0x7fa96c032ec0 comp rx=0 tx=0).stop 2024-03-17T23:55:23.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.964+0000 7fa982400640 1 -- 172.21.15.12:0/800895183 shutdown_connections 2024-03-17T23:55:23.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.964+0000 7fa982400640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa9600a2800 0x7fa9600a4cc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:23.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.964+0000 7fa982400640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97c1055b0 0x7fa97c1a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:23.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.964+0000 7fa982400640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa97c104bf0 0x7fa97c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:23.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.964+0000 7fa982400640 1 --2- 172.21.15.12:0/800895183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa97c103a00 0x7fa97c199d10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:23.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.964+0000 7fa982400640 1 -- 172.21.15.12:0/800895183 >> 172.21.15.12:0/800895183 conn(0x7fa97c0fef90 msgr2=0x7fa97c107f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:23.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.964+0000 7fa982400640 1 -- 172.21.15.12:0/800895183 shutdown_connections 2024-03-17T23:55:23.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:23.964+0000 7fa982400640 1 -- 172.21.15.12:0/800895183 wait complete. 2024-03-17T23:55:23.996 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-17T23:55:24.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.196+0000 7f0f3dc00640 1 -- 172.21.15.12:0/2647496226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f381005c0 msgr2=0x7f0f381009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:24.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.196+0000 7f0f3dc00640 1 --2- 172.21.15.12:0/2647496226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f381005c0 0x7f0f381009e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009900 tx=0x7f0f2802bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:24.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.196+0000 7f0f3dc00640 1 -- 172.21.15.12:0/2647496226 shutdown_connections 2024-03-17T23:55:24.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.196+0000 7f0f3dc00640 1 --2- 172.21.15.12:0/2647496226 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38102170 0x7f0f38108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:24.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.196+0000 7f0f3dc00640 1 --2- 172.21.15.12:0/2647496226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381017b0 0x7f0f38101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:24.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.196+0000 7f0f3dc00640 1 --2- 172.21.15.12:0/2647496226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f381005c0 0x7f0f381009e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:24.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.196+0000 7f0f3dc00640 1 -- 172.21.15.12:0/2647496226 >> 172.21.15.12:0/2647496226 conn(0x7f0f380fbb50 msgr2=0x7f0f380fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:24.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.196+0000 7f0f3dc00640 1 -- 172.21.15.12:0/2647496226 shutdown_connections 2024-03-17T23:55:24.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.196+0000 7f0f3dc00640 1 -- 172.21.15.12:0/2647496226 wait complete. 2024-03-17T23:55:24.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.197+0000 7f0f3dc00640 1 Processor -- start 2024-03-17T23:55:24.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.197+0000 7f0f3dc00640 1 -- start start 2024-03-17T23:55:24.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.197+0000 7f0f3dc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f381005c0 0x7f0f38104500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.197+0000 7f0f3dc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381017b0 0x7f0f38102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.197+0000 7f0f3dc00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38102170 0x7f0f381030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f3dc00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3810c230 con 0x7f0f381005c0 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f3dc00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3810c0b0 con 0x7f0f38102170 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f3dc00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3810c3b0 con 0x7f0f381017b0 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f37e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f381005c0 0x7f0f38104500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f37e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f381005c0 0x7f0f38104500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51036/0 (socket says 172.21.15.12:51036) 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f37e00640 1 -- 172.21.15.12:0/3318304535 learned_addr learned my addr 172.21.15.12:0/3318304535 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f3d200640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38102170 0x7f0f381030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f2fe00640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381017b0 0x7f0f38102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f3d200640 1 -- 172.21.15.12:0/3318304535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381017b0 msgr2=0x7f0f38102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f3d200640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381017b0 0x7f0f38102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f3d200640 1 -- 172.21.15.12:0/3318304535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f381005c0 msgr2=0x7f0f38104500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:24.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f3d200640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f381005c0 0x7f0f38104500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:24.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f3d200640 1 -- 172.21.15.12:0/3318304535 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f38103990 con 0x7f0f38102170 2024-03-17T23:55:24.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.198+0000 7f0f2fe00640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381017b0 0x7f0f38102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:55:24.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.199+0000 7f0f3d200640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38102170 0x7f0f381030d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009fd0 tx=0x7f0f3000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:24.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.199+0000 7f0f35600640 1 -- 172.21.15.12:0/3318304535 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3000ce20 con 0x7f0f38102170 2024-03-17T23:55:24.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.199+0000 7f0f35600640 1 -- 172.21.15.12:0/3318304535 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f30005020 con 0x7f0f38102170 2024-03-17T23:55:24.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.199+0000 7f0f35600640 1 -- 172.21.15.12:0/3318304535 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f30010720 con 0x7f0f38102170 2024-03-17T23:55:24.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.199+0000 7f0f3dc00640 1 -- 172.21.15.12:0/3318304535 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f381a1b50 con 0x7f0f38102170 2024-03-17T23:55:24.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.199+0000 7f0f3dc00640 1 -- 172.21.15.12:0/3318304535 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f381a2020 con 0x7f0f38102170 2024-03-17T23:55:24.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.200+0000 7f0f3dc00640 1 -- 172.21.15.12:0/3318304535 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f381a2250 con 0x7f0f38102170 2024-03-17T23:55:24.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.202+0000 7f0f35600640 1 -- 172.21.15.12:0/3318304535 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0f30005540 con 0x7f0f38102170 2024-03-17T23:55:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.203+0000 7f0f35600640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0f1c0a2800 0x7f0f1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.203+0000 7f0f35600640 1 -- 172.21.15.12:0/3318304535 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f0f300cf360 con 0x7f0f38102170 2024-03-17T23:55:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.204+0000 7f0f37e00640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0f1c0a2800 0x7f0f1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:24.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.204+0000 7f0f37e00640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0f1c0a2800 0x7f0f1c0a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0f2802c8e0 tx=0x7f0f28005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:24.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.207+0000 7f0f35600640 1 -- 172.21.15.12:0/3318304535 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f3009c160 con 0x7f0f38102170 2024-03-17T23:55:24.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:23 smithi079 ceph-mon[36241]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-17T23:55:24.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:23 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-17T23:55:24.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:23 smithi079 ceph-mon[36241]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:55:24.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:23 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/800895183' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-17T23:55:24.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:23 smithi191 ceph-mon[36708]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-17T23:55:24.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:23 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-17T23:55:24.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:23 smithi191 ceph-mon[36708]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:55:24.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:23 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/800895183' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-17T23:55:24.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:23 smithi012 ceph-mon[31541]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-17T23:55:24.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:23 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-17T23:55:24.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:23 smithi012 ceph-mon[31541]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:55:24.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:23 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/800895183' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-17T23:55:24.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.453+0000 7f0f3dc00640 1 -- 172.21.15.12:0/3318304535 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f0f381a25e0 con 0x7f0f38102170 2024-03-17T23:55:24.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.978+0000 7f0f35600640 1 -- 172.21.15.12:0/3318304535 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f0f3009ba60 con 0x7f0f38102170 2024-03-17T23:55:24.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.982+0000 7f0f3dc00640 1 -- 172.21.15.12:0/3318304535 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0f1c0a2800 msgr2=0x7f0f1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:24.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.982+0000 7f0f3dc00640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0f1c0a2800 0x7f0f1c0a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0f2802c8e0 tx=0x7f0f28005b50 comp rx=0 tx=0).stop 2024-03-17T23:55:24.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.982+0000 7f0f3dc00640 1 -- 172.21.15.12:0/3318304535 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38102170 msgr2=0x7f0f381030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:24.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.982+0000 7f0f3dc00640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38102170 0x7f0f381030d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009fd0 tx=0x7f0f3000ef10 comp rx=0 tx=0).stop 2024-03-17T23:55:24.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.982+0000 7f0f3dc00640 1 -- 172.21.15.12:0/3318304535 shutdown_connections 2024-03-17T23:55:24.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.982+0000 7f0f3dc00640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0f1c0a2800 0x7f0f1c0a4cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:24.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.982+0000 7f0f3dc00640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38102170 0x7f0f381030d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:24.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.982+0000 7f0f3dc00640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381017b0 0x7f0f38102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:24.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.982+0000 7f0f3dc00640 1 --2- 172.21.15.12:0/3318304535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f381005c0 0x7f0f38104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:24.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.982+0000 7f0f3dc00640 1 -- 172.21.15.12:0/3318304535 >> 172.21.15.12:0/3318304535 conn(0x7f0f380fbb50 msgr2=0x7f0f380fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:24.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.983+0000 7f0f3dc00640 1 -- 172.21.15.12:0/3318304535 shutdown_connections 2024-03-17T23:55:24.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:24.983+0000 7f0f3dc00640 1 -- 172.21.15.12:0/3318304535 wait complete. 2024-03-17T23:55:25.010 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-17T23:55:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.202+0000 7f8455800640 1 -- 172.21.15.12:0/1513039634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f84501016e0 msgr2=0x7f8450101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.202+0000 7f8455800640 1 --2- 172.21.15.12:0/1513039634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f84501016e0 0x7f8450101b00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8440009900 tx=0x7f844002bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.203+0000 7f8455800640 1 -- 172.21.15.12:0/1513039634 shutdown_connections 2024-03-17T23:55:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.203+0000 7f8455800640 1 --2- 172.21.15.12:0/1513039634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8450103290 0x7f8450109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.203+0000 7f8455800640 1 --2- 172.21.15.12:0/1513039634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84501028d0 0x7f8450102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.203+0000 7f8455800640 1 --2- 172.21.15.12:0/1513039634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f84501016e0 0x7f8450101b00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.203+0000 7f8455800640 1 -- 172.21.15.12:0/1513039634 >> 172.21.15.12:0/1513039634 conn(0x7f84500fcc50 msgr2=0x7f84500ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.203+0000 7f8455800640 1 -- 172.21.15.12:0/1513039634 shutdown_connections 2024-03-17T23:55:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.203+0000 7f8455800640 1 -- 172.21.15.12:0/1513039634 wait complete. 2024-03-17T23:55:25.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.205+0000 7f8455800640 1 Processor -- start 2024-03-17T23:55:25.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.205+0000 7f8455800640 1 -- start start 2024-03-17T23:55:25.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f8455800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84501016e0 0x7f8450075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:25.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f8455800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84501028d0 0x7f8450073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:25.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f8455800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8450103290 0x7f8450074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:25.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f8455800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f845010d350 con 0x7f84501016e0 2024-03-17T23:55:25.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f8455800640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f845010d1d0 con 0x7f8450103290 2024-03-17T23:55:25.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f8455800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f845010d4d0 con 0x7f84501028d0 2024-03-17T23:55:25.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f844fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84501016e0 0x7f8450075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:25.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f844fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84501016e0 0x7f8450075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51054/0 (socket says 172.21.15.12:51054) 2024-03-17T23:55:25.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f844fe00640 1 -- 172.21.15.12:0/3503006484 learned_addr learned my addr 172.21.15.12:0/3503006484 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f8454e00640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8450103290 0x7f8450074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f844f400640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84501028d0 0x7f8450073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f844fe00640 1 -- 172.21.15.12:0/3503006484 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84501028d0 msgr2=0x7f8450073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f844fe00640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84501028d0 0x7f8450073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f844fe00640 1 -- 172.21.15.12:0/3503006484 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8450103290 msgr2=0x7f8450074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f844fe00640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8450103290 0x7f8450074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f844fe00640 1 -- 172.21.15.12:0/3503006484 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84500749a0 con 0x7f84501016e0 2024-03-17T23:55:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.206+0000 7f8454e00640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8450103290 0x7f8450074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:55:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.207+0000 7f844fe00640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84501016e0 0x7f8450075590 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8440000c00 tx=0x7f844002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.207+0000 7f844cc00640 1 -- 172.21.15.12:0/3503006484 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440041070 con 0x7f84501016e0 2024-03-17T23:55:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.207+0000 7f844cc00640 1 -- 172.21.15.12:0/3503006484 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8440004450 con 0x7f84501016e0 2024-03-17T23:55:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.207+0000 7f844cc00640 1 -- 172.21.15.12:0/3503006484 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440034440 con 0x7f84501016e0 2024-03-17T23:55:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.207+0000 7f8455800640 1 -- 172.21.15.12:0/3503006484 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84501a1ae0 con 0x7f84501016e0 2024-03-17T23:55:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.207+0000 7f8455800640 1 -- 172.21.15.12:0/3503006484 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84501a1fe0 con 0x7f84501016e0 2024-03-17T23:55:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.209+0000 7f8455800640 1 -- 172.21.15.12:0/3503006484 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8450108aa0 con 0x7f84501016e0 2024-03-17T23:55:25.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.211+0000 7f844cc00640 1 -- 172.21.15.12:0/3503006484 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8440044070 con 0x7f84501016e0 2024-03-17T23:55:25.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.212+0000 7f844cc00640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f84340a2800 0x7f84340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:25.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.212+0000 7f844cc00640 1 -- 172.21.15.12:0/3503006484 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f84400f0870 con 0x7f84501016e0 2024-03-17T23:55:25.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.213+0000 7f844f400640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f84340a2800 0x7f84340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:25.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.214+0000 7f844f400640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f84340a2800 0x7f84340a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8450075300 tx=0x7f843c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:25.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.218+0000 7f844cc00640 1 -- 172.21.15.12:0/3503006484 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f84400bd700 con 0x7f84501016e0 2024-03-17T23:55:25.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:24 smithi079 ceph-mon[36241]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-17T23:55:25.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:24 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/800895183' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-17T23:55:25.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:24 smithi079 ceph-mon[36241]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-17T23:55:25.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:24 smithi079 ceph-mon[36241]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:55:25.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:24 smithi079 ceph-mon[36241]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-17T23:55:25.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:24 smithi079 ceph-mon[36241]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-17T23:55:25.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:24 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3318304535' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-17T23:55:25.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:24 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-17T23:55:25.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:24 smithi191 ceph-mon[36708]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-17T23:55:25.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:24 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/800895183' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-17T23:55:25.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:24 smithi191 ceph-mon[36708]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-17T23:55:25.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:24 smithi191 ceph-mon[36708]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:55:25.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:24 smithi191 ceph-mon[36708]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-17T23:55:25.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:24 smithi191 ceph-mon[36708]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-17T23:55:25.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:24 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3318304535' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-17T23:55:25.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:24 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-17T23:55:25.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:24 smithi012 ceph-mon[31541]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-17T23:55:25.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:24 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/800895183' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-17T23:55:25.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:24 smithi012 ceph-mon[31541]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-17T23:55:25.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:24 smithi012 ceph-mon[31541]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-17T23:55:25.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:24 smithi012 ceph-mon[31541]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-17T23:55:25.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:24 smithi012 ceph-mon[31541]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-17T23:55:25.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:24 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3318304535' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-17T23:55:25.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:24 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-17T23:55:25.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.465+0000 7f8455800640 1 -- 172.21.15.12:0/3503006484 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f845002d3d0 con 0x7f84501016e0 2024-03-17T23:55:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.990+0000 7f844cc00640 1 -- 172.21.15.12:0/3503006484 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f84400bd000 con 0x7f84501016e0 2024-03-17T23:55:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.994+0000 7f8455800640 1 -- 172.21.15.12:0/3503006484 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f84340a2800 msgr2=0x7f84340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.994+0000 7f8455800640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f84340a2800 0x7f84340a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8450075300 tx=0x7f843c005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.994+0000 7f8455800640 1 -- 172.21.15.12:0/3503006484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84501016e0 msgr2=0x7f8450075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.994+0000 7f8455800640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84501016e0 0x7f8450075590 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8440000c00 tx=0x7f844002c640 comp rx=0 tx=0).stop 2024-03-17T23:55:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.994+0000 7f8455800640 1 -- 172.21.15.12:0/3503006484 shutdown_connections 2024-03-17T23:55:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.994+0000 7f8455800640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f84340a2800 0x7f84340a4cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.994+0000 7f8455800640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8450103290 0x7f8450074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.994+0000 7f8455800640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84501028d0 0x7f8450073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.994+0000 7f8455800640 1 --2- 172.21.15.12:0/3503006484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84501016e0 0x7f8450075590 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.994+0000 7f8455800640 1 -- 172.21.15.12:0/3503006484 >> 172.21.15.12:0/3503006484 conn(0x7f84500fcc50 msgr2=0x7f8450105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.995+0000 7f8455800640 1 -- 172.21.15.12:0/3503006484 shutdown_connections 2024-03-17T23:55:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:25.995+0000 7f8455800640 1 -- 172.21.15.12:0/3503006484 wait complete. 2024-03-17T23:55:26.025 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-17T23:55:26.025 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-17T23:55:26.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.219+0000 7fba1ca00640 1 -- 172.21.15.12:0/864236139 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba18102290 msgr2=0x7fba18108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:26.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.219+0000 7fba1ca00640 1 --2- 172.21.15.12:0/864236139 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba18102290 0x7fba18108b20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fba10009ca0 tx=0x7fba1002c100 comp rx=0 tx=0).stop 2024-03-17T23:55:26.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.219+0000 7fba1ca00640 1 -- 172.21.15.12:0/864236139 shutdown_connections 2024-03-17T23:55:26.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.219+0000 7fba1ca00640 1 --2- 172.21.15.12:0/864236139 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba18102290 0x7fba18108b20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.219+0000 7fba1ca00640 1 --2- 172.21.15.12:0/864236139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba181018d0 0x7fba18101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.219+0000 7fba1ca00640 1 --2- 172.21.15.12:0/864236139 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba181006e0 0x7fba18100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.219+0000 7fba1ca00640 1 -- 172.21.15.12:0/864236139 >> 172.21.15.12:0/864236139 conn(0x7fba180fbc70 msgr2=0x7fba180fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:26.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.219+0000 7fba1ca00640 1 -- 172.21.15.12:0/864236139 shutdown_connections 2024-03-17T23:55:26.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.219+0000 7fba1ca00640 1 -- 172.21.15.12:0/864236139 wait complete. 2024-03-17T23:55:26.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.220+0000 7fba1ca00640 1 Processor -- start 2024-03-17T23:55:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.220+0000 7fba1ca00640 1 -- start start 2024-03-17T23:55:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba1ca00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba181006e0 0x7fba18197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba1ca00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba181018d0 0x7fba18197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba1ca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba18102290 0x7fba1819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba1ca00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fba1810d480 con 0x7fba18102290 2024-03-17T23:55:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba1ca00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fba1810d300 con 0x7fba181018d0 2024-03-17T23:55:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba1ca00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fba1810d600 con 0x7fba181006e0 2024-03-17T23:55:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba15400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba181018d0 0x7fba18197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba15400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba181018d0 0x7fba18197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:55498/0 (socket says 172.21.15.12:55498) 2024-03-17T23:55:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba15400640 1 -- 172.21.15.12:0/3614670270 learned_addr learned my addr 172.21.15.12:0/3614670270 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba15e00640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba181006e0 0x7fba18197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba16800640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba18102290 0x7fba1819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:26.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba15400640 1 -- 172.21.15.12:0/3614670270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba181006e0 msgr2=0x7fba18197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:26.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba15400640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba181006e0 0x7fba18197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba15400640 1 -- 172.21.15.12:0/3614670270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba18102290 msgr2=0x7fba1819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:26.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba15400640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba18102290 0x7fba1819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.221+0000 7fba15400640 1 -- 172.21.15.12:0/3614670270 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1819f6d0 con 0x7fba181018d0 2024-03-17T23:55:26.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.222+0000 7fba15400640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba181018d0 0x7fba18197ff0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fba0400e9f0 tx=0x7fba0400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:26.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.222+0000 7fba0ea00640 1 -- 172.21.15.12:0/3614670270 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0400cd80 con 0x7fba181018d0 2024-03-17T23:55:26.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.222+0000 7fba0ea00640 1 -- 172.21.15.12:0/3614670270 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba04010910 con 0x7fba181018d0 2024-03-17T23:55:26.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.222+0000 7fba0ea00640 1 -- 172.21.15.12:0/3614670270 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba04018990 con 0x7fba181018d0 2024-03-17T23:55:26.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.222+0000 7fba1ca00640 1 -- 172.21.15.12:0/3614670270 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba1819f9c0 con 0x7fba181018d0 2024-03-17T23:55:26.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.222+0000 7fba1ca00640 1 -- 172.21.15.12:0/3614670270 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba1819ff60 con 0x7fba181018d0 2024-03-17T23:55:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.223+0000 7fba1ca00640 1 -- 172.21.15.12:0/3614670270 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba18107a00 con 0x7fba181018d0 2024-03-17T23:55:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.225+0000 7fba0ea00640 1 -- 172.21.15.12:0/3614670270 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fba04010ac0 con 0x7fba181018d0 2024-03-17T23:55:26.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.227+0000 7fba0ea00640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb9fc0a2880 0x7fb9fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:26.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.227+0000 7fba0ea00640 1 -- 172.21.15.12:0/3614670270 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fba040cf110 con 0x7fba181018d0 2024-03-17T23:55:26.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.228+0000 7fba15e00640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb9fc0a2880 0x7fb9fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:26.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.228+0000 7fba15e00640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb9fc0a2880 0x7fb9fc0a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fba18101730 tx=0x7fba08005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.233+0000 7fba0ea00640 1 -- 172.21.15.12:0/3614670270 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fba0409b8c0 con 0x7fba181018d0 2024-03-17T23:55:26.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:26 smithi079 ceph-mon[36241]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-17T23:55:26.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:26 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-17T23:55:26.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:26 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] up:active 2024-03-17T23:55:26.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:26 smithi079 ceph-mon[36241]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-17T23:55:26.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:26 smithi079 ceph-mon[36241]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-17T23:55:26.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:26 smithi079 ceph-mon[36241]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:creating} 9 up:standby 2024-03-17T23:55:26.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:26 smithi079 ceph-mon[36241]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-17T23:55:26.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:26 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3503006484' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-17T23:55:26.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:26 smithi191 ceph-mon[36708]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-17T23:55:26.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:26 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-17T23:55:26.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:26 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] up:active 2024-03-17T23:55:26.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:26 smithi191 ceph-mon[36708]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-17T23:55:26.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:26 smithi191 ceph-mon[36708]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-17T23:55:26.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:26 smithi191 ceph-mon[36708]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:creating} 9 up:standby 2024-03-17T23:55:26.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:26 smithi191 ceph-mon[36708]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-17T23:55:26.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:26 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3503006484' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-17T23:55:26.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:26 smithi012 ceph-mon[31541]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-17T23:55:26.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:26 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-17T23:55:26.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:26 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] up:active 2024-03-17T23:55:26.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:26 smithi012 ceph-mon[31541]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-17T23:55:26.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:26 smithi012 ceph-mon[31541]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-17T23:55:26.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:26 smithi012 ceph-mon[31541]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:creating} 9 up:standby 2024-03-17T23:55:26.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:26 smithi012 ceph-mon[31541]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-17T23:55:26.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:26 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3503006484' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-17T23:55:26.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.518+0000 7fba1ca00640 1 -- 172.21.15.12:0/3614670270 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fba18061c20 con 0x7fb9fc0a2880 2024-03-17T23:55:26.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.546+0000 7fba0ea00640 1 -- 172.21.15.12:0/3614670270 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fba18061c20 con 0x7fb9fc0a2880 2024-03-17T23:55:26.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.549+0000 7fba1ca00640 1 -- 172.21.15.12:0/3614670270 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb9fc0a2880 msgr2=0x7fb9fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:26.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.549+0000 7fba1ca00640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb9fc0a2880 0x7fb9fc0a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fba18101730 tx=0x7fba08005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:26.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.549+0000 7fba1ca00640 1 -- 172.21.15.12:0/3614670270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba181018d0 msgr2=0x7fba18197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:26.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.549+0000 7fba1ca00640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba181018d0 0x7fba18197ff0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fba0400e9f0 tx=0x7fba0400eeb0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.549+0000 7fba1ca00640 1 -- 172.21.15.12:0/3614670270 shutdown_connections 2024-03-17T23:55:26.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.549+0000 7fba1ca00640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb9fc0a2880 0x7fb9fc0a4d40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.549+0000 7fba1ca00640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba18102290 0x7fba1819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.549+0000 7fba1ca00640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba181018d0 0x7fba18197ff0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.549+0000 7fba1ca00640 1 --2- 172.21.15.12:0/3614670270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba181006e0 0x7fba18197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.549+0000 7fba1ca00640 1 -- 172.21.15.12:0/3614670270 >> 172.21.15.12:0/3614670270 conn(0x7fba180fbc70 msgr2=0x7fba18104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:26.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.549+0000 7fba1ca00640 1 -- 172.21.15.12:0/3614670270 shutdown_connections 2024-03-17T23:55:26.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.549+0000 7fba1ca00640 1 -- 172.21.15.12:0/3614670270 wait complete. 2024-03-17T23:55:26.569 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-17T23:55:26.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.782+0000 7fecfc000640 1 -- 172.21.15.12:0/2839591415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fecf4101390 msgr2=0x7fecf4101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:26.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.782+0000 7fecfc000640 1 --2- 172.21.15.12:0/2839591415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fecf4101390 0x7fecf4101810 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fecec009ca0 tx=0x7fecec02c140 comp rx=0 tx=0).stop 2024-03-17T23:55:26.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.782+0000 7fecfc000640 1 -- 172.21.15.12:0/2839591415 shutdown_connections 2024-03-17T23:55:26.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.782+0000 7fecfc000640 1 --2- 172.21.15.12:0/2839591415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fecf4101390 0x7fecf4101810 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.782+0000 7fecfc000640 1 --2- 172.21.15.12:0/2839591415 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecf41009f0 0x7fecf4100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.782+0000 7fecfc000640 1 --2- 172.21.15.12:0/2839591415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf4100090 0x7fecf41004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.782+0000 7fecfc000640 1 -- 172.21.15.12:0/2839591415 >> 172.21.15.12:0/2839591415 conn(0x7fecf40fbc70 msgr2=0x7fecf40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:26.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.782+0000 7fecfc000640 1 -- 172.21.15.12:0/2839591415 shutdown_connections 2024-03-17T23:55:26.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.782+0000 7fecfc000640 1 -- 172.21.15.12:0/2839591415 wait complete. 2024-03-17T23:55:26.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.784+0000 7fecfc000640 1 Processor -- start 2024-03-17T23:55:26.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.784+0000 7fecfc000640 1 -- start start 2024-03-17T23:55:26.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.784+0000 7fecfc000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecf4100090 0x7fecf4199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:26.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.784+0000 7fecfc000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf41009f0 0x7fecf419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:26.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecfc000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fecf4101390 0x7fecf41a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:26.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecfc000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fecf410f660 con 0x7fecf41009f0 2024-03-17T23:55:26.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecfc000640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fecf410f4e0 con 0x7fecf4101390 2024-03-17T23:55:26.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecfc000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fecf410f7e0 con 0x7fecf4100090 2024-03-17T23:55:26.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecf8c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf41009f0 0x7fecf419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:26.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecf8c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf41009f0 0x7fecf419a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51096/0 (socket says 172.21.15.12:51096) 2024-03-17T23:55:26.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecf8c00640 1 -- 172.21.15.12:0/897204907 learned_addr learned my addr 172.21.15.12:0/897204907 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:26.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecf9600640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecf4100090 0x7fecf4199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:26.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecf8c00640 1 -- 172.21.15.12:0/897204907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecf4100090 msgr2=0x7fecf4199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:26.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecf8c00640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecf4100090 0x7fecf4199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecfa000640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fecf4101390 0x7fecf41a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:26.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecf8c00640 1 -- 172.21.15.12:0/897204907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fecf4101390 msgr2=0x7fecf41a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:26.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecf8c00640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fecf4101390 0x7fecf41a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:26.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecf8c00640 1 -- 172.21.15.12:0/897204907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf41a1930 con 0x7fecf41009f0 2024-03-17T23:55:26.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.785+0000 7fecfa000640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fecf4101390 0x7fecf41a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:55:26.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.786+0000 7fecf8c00640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf41009f0 0x7fecf419a260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fece000e9f0 tx=0x7fece000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:26.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.786+0000 7fecf2000640 1 -- 172.21.15.12:0/897204907 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece000cd80 con 0x7fecf41009f0 2024-03-17T23:55:26.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.786+0000 7fecf2000640 1 -- 172.21.15.12:0/897204907 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fece0005500 con 0x7fecf41009f0 2024-03-17T23:55:26.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.786+0000 7fecf2000640 1 -- 172.21.15.12:0/897204907 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece0010690 con 0x7fecf41009f0 2024-03-17T23:55:26.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.786+0000 7fecfc000640 1 -- 172.21.15.12:0/897204907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecf41a1c20 con 0x7fecf41009f0 2024-03-17T23:55:26.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.786+0000 7fecfc000640 1 -- 172.21.15.12:0/897204907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecf41a2120 con 0x7fecf41009f0 2024-03-17T23:55:26.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.788+0000 7fecfc000640 1 -- 172.21.15.12:0/897204907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecf410afc0 con 0x7fecf41009f0 2024-03-17T23:55:26.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.789+0000 7fecf2000640 1 -- 172.21.15.12:0/897204907 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fece00046c0 con 0x7fecf41009f0 2024-03-17T23:55:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.791+0000 7fecf2000640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fecdc0a2800 0x7fecdc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.791+0000 7fecf2000640 1 -- 172.21.15.12:0/897204907 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fece00cf2e0 con 0x7fecf41009f0 2024-03-17T23:55:26.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.791+0000 7fecf9600640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fecdc0a2800 0x7fecdc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:26.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.792+0000 7fecf9600640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fecdc0a2800 0x7fecdc0a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fecf419b1f0 tx=0x7fece8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:26.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:26.796+0000 7fecf2000640 1 -- 172.21.15.12:0/897204907 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fece009c0e0 con 0x7fecf41009f0 2024-03-17T23:55:26.919 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:55:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:55:26.526+0000 7fb910800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-17T23:55:26.919 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:55:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:55:26.526+0000 7fb910800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-17T23:55:26.919 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:55:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:55:26.526+0000 7fb910800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-17T23:55:26.919 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:55:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:55:26.526+0000 7fb910800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-17T23:55:26.919 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:55:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:55:26.526+0000 7fb910800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-17T23:55:27.011 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 17 23:55:26 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b[66285]: 2024-03-17T23:55:26.676+0000 7ff92867c700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-17T23:55:27.011 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 17 23:55:26 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b[66285]: 2024-03-17T23:55:26.676+0000 7ff92867c700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-17T23:55:27.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.095+0000 7fecfc000640 1 -- 172.21.15.12:0/897204907 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fecf4061c20 con 0x7fecdc0a2800 2024-03-17T23:55:27.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.124+0000 7fecf2000640 1 -- 172.21.15.12:0/897204907 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fecf4061c20 con 0x7fecdc0a2800 2024-03-17T23:55:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.126+0000 7fecfc000640 1 -- 172.21.15.12:0/897204907 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fecdc0a2800 msgr2=0x7fecdc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.126+0000 7fecfc000640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fecdc0a2800 0x7fecdc0a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fecf419b1f0 tx=0x7fece8005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.126+0000 7fecfc000640 1 -- 172.21.15.12:0/897204907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf41009f0 msgr2=0x7fecf419a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.126+0000 7fecfc000640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf41009f0 0x7fecf419a260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fece000e9f0 tx=0x7fece000edf0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.127+0000 7fecfc000640 1 -- 172.21.15.12:0/897204907 shutdown_connections 2024-03-17T23:55:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.127+0000 7fecfc000640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fecdc0a2800 0x7fecdc0a4cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.127+0000 7fecfc000640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fecf4101390 0x7fecf41a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.127+0000 7fecfc000640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecf41009f0 0x7fecf419a260 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.127+0000 7fecfc000640 1 --2- 172.21.15.12:0/897204907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecf4100090 0x7fecf4199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.127+0000 7fecfc000640 1 -- 172.21.15.12:0/897204907 >> 172.21.15.12:0/897204907 conn(0x7fecf40fbc70 msgr2=0x7fecf40fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.127+0000 7fecfc000640 1 -- 172.21.15.12:0/897204907 shutdown_connections 2024-03-17T23:55:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.127+0000 7fecfc000640 1 -- 172.21.15.12:0/897204907 wait complete. 2024-03-17T23:55:27.147 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-17T23:55:27.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:27 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3503006484' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-17T23:55:27.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:27 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] up:active 2024-03-17T23:55:27.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:27 smithi079 ceph-mon[36241]: daemon mds.i assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-17T23:55:27.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:27 smithi079 ceph-mon[36241]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 9 up:standby 2024-03-17T23:55:27.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:27 smithi079 ceph-mon[36241]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=h=up:active,3=i=up:creating} 8 up:standby 2024-03-17T23:55:27.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:27 smithi079 ceph-mon[36241]: daemon mds.i is now active in filesystem cephfs as rank 3 2024-03-17T23:55:27.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:27 smithi079 ceph-mon[36241]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-03-17T23:55:27.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:27 smithi079 ceph-mon[36241]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:27 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3503006484' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-17T23:55:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:27 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] up:active 2024-03-17T23:55:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:27 smithi191 ceph-mon[36708]: daemon mds.i assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-17T23:55:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:27 smithi191 ceph-mon[36708]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 9 up:standby 2024-03-17T23:55:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:27 smithi191 ceph-mon[36708]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=h=up:active,3=i=up:creating} 8 up:standby 2024-03-17T23:55:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:27 smithi191 ceph-mon[36708]: daemon mds.i is now active in filesystem cephfs as rank 3 2024-03-17T23:55:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:27 smithi191 ceph-mon[36708]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-03-17T23:55:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:27 smithi191 ceph-mon[36708]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:27.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.380+0000 7f1988c00640 1 -- 172.21.15.12:0/4164127589 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1984100090 msgr2=0x7f19841004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:27.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.380+0000 7f1988c00640 1 --2- 172.21.15.12:0/4164127589 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1984100090 0x7f19841004b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1974009900 tx=0x7f197402bc90 comp rx=0 tx=0).stop 2024-03-17T23:55:27.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.381+0000 7f1988c00640 1 -- 172.21.15.12:0/4164127589 shutdown_connections 2024-03-17T23:55:27.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.381+0000 7f1988c00640 1 --2- 172.21.15.12:0/4164127589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1984101390 0x7f1984101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.381+0000 7f1988c00640 1 --2- 172.21.15.12:0/4164127589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f19841009f0 0x7f1984100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.381+0000 7f1988c00640 1 --2- 172.21.15.12:0/4164127589 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1984100090 0x7f19841004b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.381+0000 7f1988c00640 1 -- 172.21.15.12:0/4164127589 >> 172.21.15.12:0/4164127589 conn(0x7f19840fbc70 msgr2=0x7f19840fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:27.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.381+0000 7f1988c00640 1 -- 172.21.15.12:0/4164127589 shutdown_connections 2024-03-17T23:55:27.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.381+0000 7f1988c00640 1 -- 172.21.15.12:0/4164127589 wait complete. 2024-03-17T23:55:27.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.382+0000 7f1988c00640 1 Processor -- start 2024-03-17T23:55:27.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.382+0000 7f1988c00640 1 -- start start 2024-03-17T23:55:27.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.382+0000 7f1988c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1984100090 0x7f1984199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:27.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.382+0000 7f1988c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19841009f0 0x7f198419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:27.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.382+0000 7f1988c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1984101390 0x7f19841a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:27.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.382+0000 7f1988c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f198410f6a0 con 0x7f1984100090 2024-03-17T23:55:27.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.382+0000 7f1988c00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f198410f520 con 0x7f19841009f0 2024-03-17T23:55:27.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.382+0000 7f1988c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f198410f820 con 0x7f1984101390 2024-03-17T23:55:27.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.382+0000 7f1981e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1984100090 0x7f1984199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:27.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.382+0000 7f1981400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19841009f0 0x7f198419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:27.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f1981e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1984100090 0x7f1984199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51126/0 (socket says 172.21.15.12:51126) 2024-03-17T23:55:27.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f1981400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19841009f0 0x7f198419a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:55544/0 (socket says 172.21.15.12:55544) 2024-03-17T23:55:27.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f1981e00640 1 -- 172.21.15.12:0/2605954384 learned_addr learned my addr 172.21.15.12:0/2605954384 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:27.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f1982800640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1984101390 0x7f19841a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:27.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f1981400640 1 -- 172.21.15.12:0/2605954384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1984101390 msgr2=0x7f19841a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:27.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f1981400640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1984101390 0x7f19841a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f1981400640 1 -- 172.21.15.12:0/2605954384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1984100090 msgr2=0x7f1984199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:27.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f1981400640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1984100090 0x7f1984199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f1981400640 1 -- 172.21.15.12:0/2605954384 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19841a1970 con 0x7f19841009f0 2024-03-17T23:55:27.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f1982800640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1984101390 0x7f19841a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:55:27.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f1981400640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19841009f0 0x7f198419a290 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f197000e9f0 tx=0x7f197000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:27.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f197aa00640 1 -- 172.21.15.12:0/2605954384 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197000ccc0 con 0x7f19841009f0 2024-03-17T23:55:27.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f197aa00640 1 -- 172.21.15.12:0/2605954384 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1970010910 con 0x7f19841009f0 2024-03-17T23:55:27.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.383+0000 7f1988c00640 1 -- 172.21.15.12:0/2605954384 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19841a1c60 con 0x7f19841009f0 2024-03-17T23:55:27.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.384+0000 7f197aa00640 1 -- 172.21.15.12:0/2605954384 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970018960 con 0x7f19841009f0 2024-03-17T23:55:27.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.384+0000 7f1988c00640 1 -- 172.21.15.12:0/2605954384 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19841a21b0 con 0x7f19841009f0 2024-03-17T23:55:27.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.385+0000 7f1988c00640 1 -- 172.21.15.12:0/2605954384 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f198410af60 con 0x7f19841009f0 2024-03-17T23:55:27.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.389+0000 7f197aa00640 1 -- 172.21.15.12:0/2605954384 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1970010ac0 con 0x7f19841009f0 2024-03-17T23:55:27.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.391+0000 7f197aa00640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19680a2880 0x7f19680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:27.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.391+0000 7f197aa00640 1 -- 172.21.15.12:0/2605954384 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f19700ceff0 con 0x7f19841009f0 2024-03-17T23:55:27.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.391+0000 7f1981e00640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19680a2880 0x7f19680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:27.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.392+0000 7f1981e00640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19680a2880 0x7f19680a4d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f197402c8e0 tx=0x7f197402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:27.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.395+0000 7f197aa00640 1 -- 172.21.15.12:0/2605954384 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f197009b7a0 con 0x7f19841009f0 2024-03-17T23:55:27.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:27 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3503006484' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-17T23:55:27.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:27 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] up:active 2024-03-17T23:55:27.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:27 smithi012 ceph-mon[31541]: daemon mds.i assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-17T23:55:27.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:27 smithi012 ceph-mon[31541]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 9 up:standby 2024-03-17T23:55:27.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:27 smithi012 ceph-mon[31541]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=h=up:active,3=i=up:creating} 8 up:standby 2024-03-17T23:55:27.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:27 smithi012 ceph-mon[31541]: daemon mds.i is now active in filesystem cephfs as rank 3 2024-03-17T23:55:27.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:27 smithi012 ceph-mon[31541]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-03-17T23:55:27.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:27 smithi012 ceph-mon[31541]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.628+0000 7f1988c00640 1 -- 172.21.15.12:0/2605954384 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f1984061cb0 con 0x7f19680a2880 2024-03-17T23:55:27.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.637+0000 7f197aa00640 1 -- 172.21.15.12:0/2605954384 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f1984061cb0 con 0x7f19680a2880 2024-03-17T23:55:27.638 INFO:teuthology.orchestra.run.smithi012.stdout:/volumes/qa/sv_0/e22828af-b032-4e19-8959-cba6e27a4fe1 2024-03-17T23:55:27.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.640+0000 7f1988c00640 1 -- 172.21.15.12:0/2605954384 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19680a2880 msgr2=0x7f19680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:27.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.640+0000 7f1988c00640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19680a2880 0x7f19680a4d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f197402c8e0 tx=0x7f197402df90 comp rx=0 tx=0).stop 2024-03-17T23:55:27.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.640+0000 7f1988c00640 1 -- 172.21.15.12:0/2605954384 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19841009f0 msgr2=0x7f198419a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:27.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.640+0000 7f1988c00640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19841009f0 0x7f198419a290 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f197000e9f0 tx=0x7f197000eeb0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.640+0000 7f1988c00640 1 -- 172.21.15.12:0/2605954384 shutdown_connections 2024-03-17T23:55:27.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.640+0000 7f1988c00640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f19680a2880 0x7f19680a4d40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.640+0000 7f1988c00640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1984101390 0x7f19841a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.640+0000 7f1988c00640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19841009f0 0x7f198419a290 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.640+0000 7f1988c00640 1 --2- 172.21.15.12:0/2605954384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1984100090 0x7f1984199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.640+0000 7f1988c00640 1 -- 172.21.15.12:0/2605954384 >> 172.21.15.12:0/2605954384 conn(0x7f19840fbc70 msgr2=0x7f19840fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:27.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.640+0000 7f1988c00640 1 -- 172.21.15.12:0/2605954384 shutdown_connections 2024-03-17T23:55:27.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.640+0000 7f1988c00640 1 -- 172.21.15.12:0/2605954384 wait complete. 2024-03-17T23:55:27.659 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-17T23:55:27.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.895+0000 7fcf87200640 1 -- 172.21.15.12:0/1026503642 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf88104b30 msgr2=0x7fcf88106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:27.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.895+0000 7fcf87200640 1 --2- 172.21.15.12:0/1026503642 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf88104b30 0x7fcf88106f20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009900 tx=0x7fcf7002bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:27.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.895+0000 7fcf87200640 1 -- 172.21.15.12:0/1026503642 shutdown_connections 2024-03-17T23:55:27.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.895+0000 7fcf87200640 1 --2- 172.21.15.12:0/1026503642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf88107460 0x7fcf88109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.895+0000 7fcf87200640 1 --2- 172.21.15.12:0/1026503642 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf88104b30 0x7fcf88106f20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.895+0000 7fcf87200640 1 --2- 172.21.15.12:0/1026503642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf88102200 0x7fcf881045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.895+0000 7fcf87200640 1 -- 172.21.15.12:0/1026503642 >> 172.21.15.12:0/1026503642 conn(0x7fcf880fbb50 msgr2=0x7fcf880fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:27.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.895+0000 7fcf87200640 1 -- 172.21.15.12:0/1026503642 shutdown_connections 2024-03-17T23:55:27.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.895+0000 7fcf87200640 1 -- 172.21.15.12:0/1026503642 wait complete. 2024-03-17T23:55:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.896+0000 7fcf87200640 1 Processor -- start 2024-03-17T23:55:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.897+0000 7fcf87200640 1 -- start start 2024-03-17T23:55:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.897+0000 7fcf87200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf88102200 0x7fcf88199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.897+0000 7fcf87200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf88104b30 0x7fcf8819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.897+0000 7fcf87200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf88107460 0x7fcf881a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.897+0000 7fcf87200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8810d390 con 0x7fcf88104b30 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.897+0000 7fcf87200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8810d210 con 0x7fcf88102200 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.897+0000 7fcf87200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8810d510 con 0x7fcf88107460 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.897+0000 7fcf85e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf88102200 0x7fcf88199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.897+0000 7fcf85e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf88102200 0x7fcf88199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:55568/0 (socket says 172.21.15.12:55568) 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.897+0000 7fcf7fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf88104b30 0x7fcf8819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.897+0000 7fcf85e00640 1 -- 172.21.15.12:0/3466367417 learned_addr learned my addr 172.21.15.12:0/3466367417 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.897+0000 7fcf86800640 1 --2- 172.21.15.12:0/3466367417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf88107460 0x7fcf881a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.898+0000 7fcf85e00640 1 -- 172.21.15.12:0/3466367417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf88107460 msgr2=0x7fcf881a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.898+0000 7fcf85e00640 1 --2- 172.21.15.12:0/3466367417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf88107460 0x7fcf881a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.898+0000 7fcf85e00640 1 -- 172.21.15.12:0/3466367417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf88104b30 msgr2=0x7fcf8819a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.898+0000 7fcf85e00640 1 --2- 172.21.15.12:0/3466367417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf88104b30 0x7fcf8819a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.898+0000 7fcf85e00640 1 -- 172.21.15.12:0/3466367417 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf881a1850 con 0x7fcf88102200 2024-03-17T23:55:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.898+0000 7fcf85e00640 1 --2- 172.21.15.12:0/3466367417 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf88102200 0x7fcf88199c40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800e9f0 tx=0x7fcf7800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.898+0000 7fcf7ea00640 1 -- 172.21.15.12:0/3466367417 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7800cd80 con 0x7fcf88102200 2024-03-17T23:55:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.898+0000 7fcf7ea00640 1 -- 172.21.15.12:0/3466367417 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf78005500 con 0x7fcf88102200 2024-03-17T23:55:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.898+0000 7fcf87200640 1 -- 172.21.15.12:0/3466367417 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf881a1b40 con 0x7fcf88102200 2024-03-17T23:55:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.898+0000 7fcf7ea00640 1 -- 172.21.15.12:0/3466367417 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78010690 con 0x7fcf88102200 2024-03-17T23:55:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.899+0000 7fcf87200640 1 -- 172.21.15.12:0/3466367417 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf881a2090 con 0x7fcf88102200 2024-03-17T23:55:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.900+0000 7fcf87200640 1 -- 172.21.15.12:0/3466367417 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf88108f50 con 0x7fcf88102200 2024-03-17T23:55:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.901+0000 7fcf7ea00640 1 -- 172.21.15.12:0/3466367417 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcf780046c0 con 0x7fcf88102200 2024-03-17T23:55:27.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.903+0000 7fcf7ea00640 1 --2- 172.21.15.12:0/3466367417 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcf6c0a2880 0x7fcf6c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:27.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.903+0000 7fcf7ea00640 1 -- 172.21.15.12:0/3466367417 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fcf780cf0f0 con 0x7fcf88102200 2024-03-17T23:55:27.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.903+0000 7fcf7fe00640 1 --2- 172.21.15.12:0/3466367417 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcf6c0a2880 0x7fcf6c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:27.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.904+0000 7fcf7fe00640 1 --2- 172.21.15.12:0/3466367417 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcf6c0a2880 0x7fcf6c0a4d40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcf7002c170 tx=0x7fcf7002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:27.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:27.906+0000 7fcf7ea00640 1 -- 172.21.15.12:0/3466367417 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcf7809bef0 con 0x7fcf88102200 2024-03-17T23:55:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.189+0000 7fcf87200640 1 -- 172.21.15.12:0/3466367417 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fcf88061c20 con 0x7fcf6c0a2880 2024-03-17T23:55:28.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.221+0000 7fcf7ea00640 1 -- 172.21.15.12:0/3466367417 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcf88061c20 con 0x7fcf6c0a2880 2024-03-17T23:55:28.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.224+0000 7fcf87200640 1 -- 172.21.15.12:0/3466367417 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcf6c0a2880 msgr2=0x7fcf6c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:28.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.224+0000 7fcf87200640 1 --2- 172.21.15.12:0/3466367417 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcf6c0a2880 0x7fcf6c0a4d40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcf7002c170 tx=0x7fcf7002df90 comp rx=0 tx=0).stop 2024-03-17T23:55:28.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.224+0000 7fcf87200640 1 -- 172.21.15.12:0/3466367417 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf88102200 msgr2=0x7fcf88199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:28.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.224+0000 7fcf87200640 1 --2- 172.21.15.12:0/3466367417 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf88102200 0x7fcf88199c40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800e9f0 tx=0x7fcf7800edf0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.224+0000 7fcf87200640 1 -- 172.21.15.12:0/3466367417 shutdown_connections 2024-03-17T23:55:28.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.224+0000 7fcf87200640 1 --2- 172.21.15.12:0/3466367417 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcf6c0a2880 0x7fcf6c0a4d40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.224+0000 7fcf87200640 1 --2- 172.21.15.12:0/3466367417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf88107460 0x7fcf881a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.224+0000 7fcf87200640 1 --2- 172.21.15.12:0/3466367417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf88104b30 0x7fcf8819a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.224+0000 7fcf87200640 1 --2- 172.21.15.12:0/3466367417 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf88102200 0x7fcf88199c40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.224+0000 7fcf87200640 1 -- 172.21.15.12:0/3466367417 >> 172.21.15.12:0/3466367417 conn(0x7fcf880fbb50 msgr2=0x7fcf88101970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:28.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.225+0000 7fcf87200640 1 -- 172.21.15.12:0/3466367417 shutdown_connections 2024-03-17T23:55:28.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.225+0000 7fcf87200640 1 -- 172.21.15.12:0/3466367417 wait complete. 2024-03-17T23:55:28.245 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-17T23:55:28.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:28 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] up:active 2024-03-17T23:55:28.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:28 smithi079 ceph-mon[36241]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-17T23:55:28.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:28 smithi079 ceph-mon[36241]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-17T23:55:28.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:28 smithi079 ceph-mon[36241]: Cluster is now healthy 2024-03-17T23:55:28.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:28 smithi079 ceph-mon[36241]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=h=up:active,3=i=up:active} 8 up:standby 2024-03-17T23:55:28.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:28 smithi079 ceph-mon[36241]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-17T23:55:28.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:28 smithi079 ceph-mon[36241]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-03-17T23:55:28.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:28 smithi079 ceph-mon[36241]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:28.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:28 smithi079 ceph-mon[36241]: from='client.24842 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:28.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:28 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] up:active 2024-03-17T23:55:28.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:28 smithi012 ceph-mon[31541]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-17T23:55:28.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:28 smithi012 ceph-mon[31541]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-17T23:55:28.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:28 smithi012 ceph-mon[31541]: Cluster is now healthy 2024-03-17T23:55:28.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:28 smithi012 ceph-mon[31541]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=h=up:active,3=i=up:active} 8 up:standby 2024-03-17T23:55:28.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:28 smithi012 ceph-mon[31541]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-17T23:55:28.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:28 smithi012 ceph-mon[31541]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-03-17T23:55:28.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:28 smithi012 ceph-mon[31541]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:28.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:28 smithi012 ceph-mon[31541]: from='client.24842 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:28.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:28 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] up:active 2024-03-17T23:55:28.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:28 smithi191 ceph-mon[36708]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-17T23:55:28.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:28 smithi191 ceph-mon[36708]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-17T23:55:28.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:28 smithi191 ceph-mon[36708]: Cluster is now healthy 2024-03-17T23:55:28.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:28 smithi191 ceph-mon[36708]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=h=up:active,3=i=up:active} 8 up:standby 2024-03-17T23:55:28.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:28 smithi191 ceph-mon[36708]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-17T23:55:28.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:28 smithi191 ceph-mon[36708]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-03-17T23:55:28.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:28 smithi191 ceph-mon[36708]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:28.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:28 smithi191 ceph-mon[36708]: from='client.24842 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:28.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.487+0000 7f8f25400640 1 -- 172.21.15.12:0/3216698625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f20102160 msgr2=0x7f8f20104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:28.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.487+0000 7f8f25400640 1 --2- 172.21.15.12:0/3216698625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f20102160 0x7f8f20104550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009900 tx=0x7f8f1002bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:28.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.487+0000 7f8f25400640 1 -- 172.21.15.12:0/3216698625 shutdown_connections 2024-03-17T23:55:28.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.487+0000 7f8f25400640 1 --2- 172.21.15.12:0/3216698625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f201073c0 0x7f8f20109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.487+0000 7f8f25400640 1 --2- 172.21.15.12:0/3216698625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f20104a90 0x7f8f20106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.487+0000 7f8f25400640 1 --2- 172.21.15.12:0/3216698625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f20102160 0x7f8f20104550 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.487+0000 7f8f25400640 1 -- 172.21.15.12:0/3216698625 >> 172.21.15.12:0/3216698625 conn(0x7f8f200fbb50 msgr2=0x7f8f200fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:28.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.487+0000 7f8f25400640 1 -- 172.21.15.12:0/3216698625 shutdown_connections 2024-03-17T23:55:28.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.487+0000 7f8f25400640 1 -- 172.21.15.12:0/3216698625 wait complete. 2024-03-17T23:55:28.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.489+0000 7f8f25400640 1 Processor -- start 2024-03-17T23:55:28.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.489+0000 7f8f25400640 1 -- start start 2024-03-17T23:55:28.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f25400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f20102160 0x7f8f20199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:28.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f25400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f20104a90 0x7f8f2019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:28.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f25400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f201073c0 0x7f8f201a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f25400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2010d3b0 con 0x7f8f201073c0 2024-03-17T23:55:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f25400640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2010d230 con 0x7f8f20104a90 2024-03-17T23:55:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f25400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2010d530 con 0x7f8f20102160 2024-03-17T23:55:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f17e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f20104a90 0x7f8f2019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f1fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f20102160 0x7f8f20199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f17e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f20104a90 0x7f8f2019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:55588/0 (socket says 172.21.15.12:55588) 2024-03-17T23:55:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f17e00640 1 -- 172.21.15.12:0/361493825 learned_addr learned my addr 172.21.15.12:0/361493825 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f24a00640 1 --2- 172.21.15.12:0/361493825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f201073c0 0x7f8f201a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:28.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f17e00640 1 -- 172.21.15.12:0/361493825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f20102160 msgr2=0x7f8f20199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:28.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f17e00640 1 --2- 172.21.15.12:0/361493825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f20102160 0x7f8f20199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f17e00640 1 -- 172.21.15.12:0/361493825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f201073c0 msgr2=0x7f8f201a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:28.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f17e00640 1 --2- 172.21.15.12:0/361493825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f201073c0 0x7f8f201a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.490+0000 7f8f17e00640 1 -- 172.21.15.12:0/361493825 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f201a1850 con 0x7f8f20104a90 2024-03-17T23:55:28.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.491+0000 7f8f17e00640 1 --2- 172.21.15.12:0/361493825 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f20104a90 0x7f8f2019a180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8f0800e9c0 tx=0x7f8f0800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:28.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.491+0000 7f8f1d600640 1 -- 172.21.15.12:0/361493825 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f0800cd80 con 0x7f8f20104a90 2024-03-17T23:55:28.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.491+0000 7f8f1d600640 1 -- 172.21.15.12:0/361493825 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f08005500 con 0x7f8f20104a90 2024-03-17T23:55:28.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.491+0000 7f8f25400640 1 -- 172.21.15.12:0/361493825 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f201a1b40 con 0x7f8f20104a90 2024-03-17T23:55:28.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.491+0000 7f8f1d600640 1 -- 172.21.15.12:0/361493825 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f08010690 con 0x7f8f20104a90 2024-03-17T23:55:28.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.491+0000 7f8f25400640 1 -- 172.21.15.12:0/361493825 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f201a2060 con 0x7f8f20104a90 2024-03-17T23:55:28.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.492+0000 7f8f25400640 1 -- 172.21.15.12:0/361493825 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f20108f10 con 0x7f8f20104a90 2024-03-17T23:55:28.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.494+0000 7f8f1d600640 1 -- 172.21.15.12:0/361493825 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8f080046c0 con 0x7f8f20104a90 2024-03-17T23:55:28.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.495+0000 7f8f1d600640 1 --2- 172.21.15.12:0/361493825 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8f040a2800 0x7f8f040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:28.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.496+0000 7f8f1d600640 1 -- 172.21.15.12:0/361493825 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8f080cf0f0 con 0x7f8f20104a90 2024-03-17T23:55:28.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.496+0000 7f8f1fe00640 1 --2- 172.21.15.12:0/361493825 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8f040a2800 0x7f8f040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:28.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.497+0000 7f8f1fe00640 1 --2- 172.21.15.12:0/361493825 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8f040a2800 0x7f8f040a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8f1002c240 tx=0x7f8f1002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:28.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.499+0000 7f8f1d600640 1 -- 172.21.15.12:0/361493825 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8f0809bef0 con 0x7f8f20104a90 2024-03-17T23:55:28.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.732+0000 7f8f25400640 1 -- 172.21.15.12:0/361493825 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f8f20061c20 con 0x7f8f040a2800 2024-03-17T23:55:28.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.740+0000 7f8f1d600640 1 -- 172.21.15.12:0/361493825 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f8f20061c20 con 0x7f8f040a2800 2024-03-17T23:55:28.741 INFO:teuthology.orchestra.run.smithi012.stdout:/volumes/qa/sv_1/7f0ea4d6-29cb-4f0b-8d34-75c93382a378 2024-03-17T23:55:28.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.743+0000 7f8f25400640 1 -- 172.21.15.12:0/361493825 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8f040a2800 msgr2=0x7f8f040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:28.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.743+0000 7f8f25400640 1 --2- 172.21.15.12:0/361493825 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8f040a2800 0x7f8f040a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8f1002c240 tx=0x7f8f1002df90 comp rx=0 tx=0).stop 2024-03-17T23:55:28.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.743+0000 7f8f25400640 1 -- 172.21.15.12:0/361493825 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f20104a90 msgr2=0x7f8f2019a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:28.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.743+0000 7f8f25400640 1 --2- 172.21.15.12:0/361493825 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f20104a90 0x7f8f2019a180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8f0800e9c0 tx=0x7f8f0800ee80 comp rx=0 tx=0).stop 2024-03-17T23:55:28.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.743+0000 7f8f25400640 1 -- 172.21.15.12:0/361493825 shutdown_connections 2024-03-17T23:55:28.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.743+0000 7f8f25400640 1 --2- 172.21.15.12:0/361493825 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8f040a2800 0x7f8f040a4cc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.743+0000 7f8f25400640 1 --2- 172.21.15.12:0/361493825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f201073c0 0x7f8f201a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.743+0000 7f8f25400640 1 --2- 172.21.15.12:0/361493825 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f20104a90 0x7f8f2019a180 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.743+0000 7f8f25400640 1 --2- 172.21.15.12:0/361493825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f20102160 0x7f8f20199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.743+0000 7f8f25400640 1 -- 172.21.15.12:0/361493825 >> 172.21.15.12:0/361493825 conn(0x7f8f200fbb50 msgr2=0x7f8f20101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:28.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.743+0000 7f8f25400640 1 -- 172.21.15.12:0/361493825 shutdown_connections 2024-03-17T23:55:28.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.743+0000 7f8f25400640 1 -- 172.21.15.12:0/361493825 wait complete. 2024-03-17T23:55:28.762 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:55:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.968+0000 7f96a9c00640 1 -- 172.21.15.12:0/1418143331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96a4103390 msgr2=0x7f96a4109c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.968+0000 7f96a9c00640 1 --2- 172.21.15.12:0/1418143331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96a4103390 0x7f96a4109c20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f969c009ca0 tx=0x7f969c02c0f0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.968+0000 7f96a9c00640 1 -- 172.21.15.12:0/1418143331 shutdown_connections 2024-03-17T23:55:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.968+0000 7f96a9c00640 1 --2- 172.21.15.12:0/1418143331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96a4103390 0x7f96a4109c20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.968+0000 7f96a9c00640 1 --2- 172.21.15.12:0/1418143331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96a41029d0 0x7f96a4102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.968+0000 7f96a9c00640 1 --2- 172.21.15.12:0/1418143331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96a41017e0 0x7f96a4101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.968+0000 7f96a9c00640 1 -- 172.21.15.12:0/1418143331 >> 172.21.15.12:0/1418143331 conn(0x7f96a40fcd50 msgr2=0x7f96a40ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.968+0000 7f96a9c00640 1 -- 172.21.15.12:0/1418143331 shutdown_connections 2024-03-17T23:55:28.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.968+0000 7f96a9c00640 1 -- 172.21.15.12:0/1418143331 wait complete. 2024-03-17T23:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.969+0000 7f96a9c00640 1 Processor -- start 2024-03-17T23:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.970+0000 7f96a9c00640 1 -- start start 2024-03-17T23:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.970+0000 7f96a9c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96a41017e0 0x7f96a4199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.970+0000 7f96a9c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96a41029d0 0x7f96a419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.970+0000 7f96a9c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96a4103390 0x7f96a41a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.970+0000 7f96a9c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f96a410d440 con 0x7f96a4103390 2024-03-17T23:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.970+0000 7f96a9c00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f96a410d2c0 con 0x7f96a41029d0 2024-03-17T23:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.970+0000 7f96a9c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f96a410d5c0 con 0x7f96a41017e0 2024-03-17T23:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.970+0000 7f96a2000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96a41029d0 0x7f96a419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.970+0000 7f96a2a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96a41017e0 0x7f96a4199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.970+0000 7f96a2000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96a41029d0 0x7f96a419a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:55610/0 (socket says 172.21.15.12:55610) 2024-03-17T23:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.970+0000 7f96a2000640 1 -- 172.21.15.12:0/2158995707 learned_addr learned my addr 172.21.15.12:0/2158995707 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.970+0000 7f96a3400640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96a4103390 0x7f96a41a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.971+0000 7f96a2000640 1 -- 172.21.15.12:0/2158995707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96a41017e0 msgr2=0x7f96a4199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.971+0000 7f96a2000640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96a41017e0 0x7f96a4199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.971+0000 7f96a2000640 1 -- 172.21.15.12:0/2158995707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96a4103390 msgr2=0x7f96a41a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.971+0000 7f96a2000640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96a4103390 0x7f96a41a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.971+0000 7f96a2000640 1 -- 172.21.15.12:0/2158995707 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a41a1940 con 0x7f96a41029d0 2024-03-17T23:55:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.971+0000 7f96a2a00640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96a41017e0 0x7f96a4199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:55:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.971+0000 7f96a2000640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96a41029d0 0x7f96a419a270 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f969000e9f0 tx=0x7f969000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.971+0000 7f969b400640 1 -- 172.21.15.12:0/2158995707 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969000cd80 con 0x7f96a41029d0 2024-03-17T23:55:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.971+0000 7f969b400640 1 -- 172.21.15.12:0/2158995707 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9690010910 con 0x7f96a41029d0 2024-03-17T23:55:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.971+0000 7f96a9c00640 1 -- 172.21.15.12:0/2158995707 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96a41a1c30 con 0x7f96a41029d0 2024-03-17T23:55:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.971+0000 7f969b400640 1 -- 172.21.15.12:0/2158995707 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690018990 con 0x7f96a41029d0 2024-03-17T23:55:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.971+0000 7f96a9c00640 1 -- 172.21.15.12:0/2158995707 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96a41a21d0 con 0x7f96a41029d0 2024-03-17T23:55:28.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.972+0000 7f96a9c00640 1 -- 172.21.15.12:0/2158995707 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96a41a23a0 con 0x7f96a41029d0 2024-03-17T23:55:28.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.977+0000 7f969b400640 1 -- 172.21.15.12:0/2158995707 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9690010ac0 con 0x7f96a41029d0 2024-03-17T23:55:28.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.978+0000 7f969b400640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f96880a2880 0x7f96880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.978+0000 7f969b400640 1 -- 172.21.15.12:0/2158995707 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f96900cef50 con 0x7f96a41029d0 2024-03-17T23:55:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.978+0000 7f96a2a00640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f96880a2880 0x7f96880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.979+0000 7f96a2a00640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f96880a2880 0x7f96880a4d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f96a419b200 tx=0x7f9694005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:28.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:28.982+0000 7f969b400640 1 -- 172.21.15.12:0/2158995707 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f969009b760 con 0x7f96a41029d0 2024-03-17T23:55:29.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.057+0000 7f969b400640 1 -- 172.21.15.12:0/2158995707 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9690097d40 con 0x7f96a41029d0 2024-03-17T23:55:29.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:29 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] up:active 2024-03-17T23:55:29.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:29 smithi079 ceph-mon[36241]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-17T23:55:29.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:29 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:55:29.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:29 smithi079 ceph-mon[36241]: pgmap v337: 129 pgs: 129 active+clean; 584 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 5 op/s 2024-03-17T23:55:29.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:29 smithi079 ceph-mon[36241]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:29.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:29 smithi079 ceph-mon[36241]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:29.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.300+0000 7f96a9c00640 1 -- 172.21.15.12:0/2158995707 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96a41a2730 con 0x7f96a41029d0 2024-03-17T23:55:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.301+0000 7f969b400640 1 -- 172.21.15.12:0/2158995707 <== mon.2 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11936 (secure 0 0 0) 0x7f96900a1d80 con 0x7f96a41029d0 2024-03-17T23:55:29.302 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:29.302 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-17T23:54:56.312554+0000","modified":"2024-03-17T23:55:28.030481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24575,"mds_2":24530,"mds_3":24529,"mds_4":24514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.12:6841/2073607248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":2073607248},{"type":"v1","addr":"172.21.15.12:6841","nonce":2073607248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/2478742711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":2478742711},{"type":"v1","addr":"172.21.15.79:6839","nonce":2478742711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24575,24587,24530,24529,24514]},"id":1}]} 2024-03-17T23:55:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 28 2024-03-17T23:55:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.303+0000 7f96a9c00640 1 -- 172.21.15.12:0/2158995707 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f96880a2880 msgr2=0x7f96880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.304+0000 7f96a9c00640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f96880a2880 0x7f96880a4d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f96a419b200 tx=0x7f9694005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.304+0000 7f96a9c00640 1 -- 172.21.15.12:0/2158995707 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96a41029d0 msgr2=0x7f96a419a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.304+0000 7f96a9c00640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96a41029d0 0x7f96a419a270 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f969000e9f0 tx=0x7f969000eeb0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.304+0000 7f96a9c00640 1 -- 172.21.15.12:0/2158995707 shutdown_connections 2024-03-17T23:55:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.304+0000 7f96a9c00640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f96880a2880 0x7f96880a4d40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.304+0000 7f96a9c00640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96a4103390 0x7f96a41a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.304+0000 7f96a9c00640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96a41029d0 0x7f96a419a270 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.304+0000 7f96a9c00640 1 --2- 172.21.15.12:0/2158995707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96a41017e0 0x7f96a4199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.304+0000 7f96a9c00640 1 -- 172.21.15.12:0/2158995707 >> 172.21.15.12:0/2158995707 conn(0x7f96a40fcd50 msgr2=0x7f96a4073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.304+0000 7f96a9c00640 1 -- 172.21.15.12:0/2158995707 shutdown_connections 2024-03-17T23:55:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.304+0000 7f96a9c00640 1 -- 172.21.15.12:0/2158995707 wait complete. 2024-03-17T23:55:29.321 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:29 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] up:active 2024-03-17T23:55:29.321 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:29 smithi012 ceph-mon[31541]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-17T23:55:29.321 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:29 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:55:29.321 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:29 smithi012 ceph-mon[31541]: pgmap v337: 129 pgs: 129 active+clean; 584 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 5 op/s 2024-03-17T23:55:29.321 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:29 smithi012 ceph-mon[31541]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:29.322 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:29 smithi012 ceph-mon[31541]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:29.324 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:55:29.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:29 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] up:active 2024-03-17T23:55:29.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:29 smithi191 ceph-mon[36708]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-17T23:55:29.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:29 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:55:29.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:29 smithi191 ceph-mon[36708]: pgmap v337: 129 pgs: 129 active+clean; 584 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 5 op/s 2024-03-17T23:55:29.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:29 smithi191 ceph-mon[36708]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:29.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:29 smithi191 ceph-mon[36708]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:55:29.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.540+0000 7f0d41a00640 1 -- 172.21.15.12:0/260494005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d3c1017f0 msgr2=0x7f0d3c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:29.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.540+0000 7f0d41a00640 1 --2- 172.21.15.12:0/260494005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d3c1017f0 0x7f0d3c101c10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c009900 tx=0x7f0d2c02bc50 comp rx=0 tx=0).stop 2024-03-17T23:55:29.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.540+0000 7f0d41a00640 1 -- 172.21.15.12:0/260494005 shutdown_connections 2024-03-17T23:55:29.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.540+0000 7f0d41a00640 1 --2- 172.21.15.12:0/260494005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d3c1033a0 0x7f0d3c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.540+0000 7f0d41a00640 1 --2- 172.21.15.12:0/260494005 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c1029e0 0x7f0d3c102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.540+0000 7f0d41a00640 1 --2- 172.21.15.12:0/260494005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d3c1017f0 0x7f0d3c101c10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.540+0000 7f0d41a00640 1 -- 172.21.15.12:0/260494005 >> 172.21.15.12:0/260494005 conn(0x7f0d3c0fcd80 msgr2=0x7f0d3c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.540+0000 7f0d41a00640 1 -- 172.21.15.12:0/260494005 shutdown_connections 2024-03-17T23:55:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.540+0000 7f0d41a00640 1 -- 172.21.15.12:0/260494005 wait complete. 2024-03-17T23:55:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.541+0000 7f0d41a00640 1 Processor -- start 2024-03-17T23:55:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.541+0000 7f0d41a00640 1 -- start start 2024-03-17T23:55:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.541+0000 7f0d41a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d3c1017f0 0x7f0d3c075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.541+0000 7f0d41a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c1029e0 0x7f0d3c073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.541+0000 7f0d41a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d3c1033a0 0x7f0d3c074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.541+0000 7f0d41a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c10d440 con 0x7f0d3c1033a0 2024-03-17T23:55:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.541+0000 7f0d41a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c10d2c0 con 0x7f0d3c1029e0 2024-03-17T23:55:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.541+0000 7f0d41a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c10d5c0 con 0x7f0d3c1017f0 2024-03-17T23:55:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.541+0000 7f0d3aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d3c1017f0 0x7f0d3c075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.542+0000 7f0d3aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d3c1017f0 0x7f0d3c075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:34540/0 (socket says 172.21.15.12:34540) 2024-03-17T23:55:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.542+0000 7f0d32000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c1029e0 0x7f0d3c073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.542+0000 7f0d3aa00640 1 -- 172.21.15.12:0/3768899117 learned_addr learned my addr 172.21.15.12:0/3768899117 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.542+0000 7f0d3b400640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d3c1033a0 0x7f0d3c074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.542+0000 7f0d3aa00640 1 -- 172.21.15.12:0/3768899117 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c1029e0 msgr2=0x7f0d3c073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.542+0000 7f0d3aa00640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c1029e0 0x7f0d3c073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.542+0000 7f0d3aa00640 1 -- 172.21.15.12:0/3768899117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d3c1033a0 msgr2=0x7f0d3c074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.542+0000 7f0d3aa00640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d3c1033a0 0x7f0d3c074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.542+0000 7f0d3aa00640 1 -- 172.21.15.12:0/3768899117 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3c0749a0 con 0x7f0d3c1017f0 2024-03-17T23:55:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.542+0000 7f0d3b400640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d3c1033a0 0x7f0d3c074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:55:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.542+0000 7f0d3aa00640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d3c1017f0 0x7f0d3c075590 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c000c00 tx=0x7f0d2c02c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.542+0000 7f0d33e00640 1 -- 172.21.15.12:0/3768899117 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c042020 con 0x7f0d3c1017f0 2024-03-17T23:55:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.543+0000 7f0d33e00640 1 -- 172.21.15.12:0/3768899117 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d2c004450 con 0x7f0d3c1017f0 2024-03-17T23:55:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.543+0000 7f0d33e00640 1 -- 172.21.15.12:0/3768899117 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c03e460 con 0x7f0d3c1017f0 2024-03-17T23:55:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.543+0000 7f0d41a00640 1 -- 172.21.15.12:0/3768899117 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d3c1a1ca0 con 0x7f0d3c1017f0 2024-03-17T23:55:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.543+0000 7f0d41a00640 1 -- 172.21.15.12:0/3768899117 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d3c1a21a0 con 0x7f0d3c1017f0 2024-03-17T23:55:29.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.544+0000 7f0d41a00640 1 -- 172.21.15.12:0/3768899117 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d3c108b10 con 0x7f0d3c1017f0 2024-03-17T23:55:29.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.546+0000 7f0d33e00640 1 -- 172.21.15.12:0/3768899117 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f0d2c045020 con 0x7f0d3c1017f0 2024-03-17T23:55:29.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.547+0000 7f0d33e00640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0d200a2890 0x7f0d200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:29.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.547+0000 7f0d33e00640 1 -- 172.21.15.12:0/3768899117 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f0d2c0f0d40 con 0x7f0d3c1017f0 2024-03-17T23:55:29.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.547+0000 7f0d32000640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0d200a2890 0x7f0d200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:29.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.548+0000 7f0d32000640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0d200a2890 0x7f0d200a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c075300 tx=0x7f0d24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.551+0000 7f0d33e00640 1 -- 172.21.15.12:0/3768899117 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0d2c04c020 con 0x7f0d3c1017f0 2024-03-17T23:55:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.803+0000 7f0d41a00640 1 -- 172.21.15.12:0/3768899117 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d3c02d420 con 0x7f0d3c1017f0 2024-03-17T23:55:29.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.805+0000 7f0d33e00640 1 -- 172.21.15.12:0/3768899117 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24044 (secure 0 0 0) 0x7f0d2c0bdb80 con 0x7f0d3c1017f0 2024-03-17T23:55:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:29.806 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":90,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:55:02.337441+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:55:29.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.808+0000 7f0d41a00640 1 -- 172.21.15.12:0/3768899117 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0d200a2890 msgr2=0x7f0d200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:29.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.808+0000 7f0d41a00640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0d200a2890 0x7f0d200a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c075300 tx=0x7f0d24005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:29.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.808+0000 7f0d41a00640 1 -- 172.21.15.12:0/3768899117 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d3c1017f0 msgr2=0x7f0d3c075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:29.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.808+0000 7f0d41a00640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d3c1017f0 0x7f0d3c075590 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c000c00 tx=0x7f0d2c02c660 comp rx=0 tx=0).stop 2024-03-17T23:55:29.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.808+0000 7f0d41a00640 1 -- 172.21.15.12:0/3768899117 shutdown_connections 2024-03-17T23:55:29.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.808+0000 7f0d41a00640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0d200a2890 0x7f0d200a4d50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.808+0000 7f0d41a00640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d3c1033a0 0x7f0d3c074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.808+0000 7f0d41a00640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c1029e0 0x7f0d3c073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.808+0000 7f0d41a00640 1 --2- 172.21.15.12:0/3768899117 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d3c1017f0 0x7f0d3c075590 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:29.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.808+0000 7f0d41a00640 1 -- 172.21.15.12:0/3768899117 >> 172.21.15.12:0/3768899117 conn(0x7f0d3c0fcd80 msgr2=0x7f0d3c105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:29.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.808+0000 7f0d41a00640 1 -- 172.21.15.12:0/3768899117 shutdown_connections 2024-03-17T23:55:29.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:29.808+0000 7f0d41a00640 1 -- 172.21.15.12:0/3768899117 wait complete. 2024-03-17T23:55:29.828 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-17T23:55:29.828 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:55:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.028+0000 7f4946a00640 1 -- 172.21.15.12:0/4043643296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4940102820 msgr2=0x7f4940102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.028+0000 7f4946a00640 1 --2- 172.21.15.12:0/4043643296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4940102820 0x7f4940102ca0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f492c009900 tx=0x7f492c02bc40 comp rx=0 tx=0).stop 2024-03-17T23:55:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.028+0000 7f4946a00640 1 -- 172.21.15.12:0/4043643296 shutdown_connections 2024-03-17T23:55:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.028+0000 7f4946a00640 1 --2- 172.21.15.12:0/4043643296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49401031e0 0x7f4940109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.028+0000 7f4946a00640 1 --2- 172.21.15.12:0/4043643296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4940102820 0x7f4940102ca0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.028+0000 7f4946a00640 1 --2- 172.21.15.12:0/4043643296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4940101630 0x7f4940101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.028+0000 7f4946a00640 1 -- 172.21.15.12:0/4043643296 >> 172.21.15.12:0/4043643296 conn(0x7f49400fcbc0 msgr2=0x7f49400fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.029+0000 7f4946a00640 1 -- 172.21.15.12:0/4043643296 shutdown_connections 2024-03-17T23:55:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.029+0000 7f4946a00640 1 -- 172.21.15.12:0/4043643296 wait complete. 2024-03-17T23:55:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.029+0000 7f4946a00640 1 Processor -- start 2024-03-17T23:55:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.030+0000 7f4946a00640 1 -- start start 2024-03-17T23:55:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.030+0000 7f4946a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4940101630 0x7f4940199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.030+0000 7f4946a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4940102820 0x7f4940199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.030+0000 7f4946a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49401031e0 0x7f49401a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.030+0000 7f4946a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f494010d2e0 con 0x7f4940101630 2024-03-17T23:55:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.030+0000 7f4946a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f494010d160 con 0x7f49401031e0 2024-03-17T23:55:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.030+0000 7f4946a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f494010d460 con 0x7f4940102820 2024-03-17T23:55:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.030+0000 7f4944c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4940102820 0x7f4940199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.030+0000 7f4945600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4940101630 0x7f4940199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.030+0000 7f4946000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49401031e0 0x7f49401a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.030+0000 7f4945600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4940101630 0x7f4940199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51232/0 (socket says 172.21.15.12:51232) 2024-03-17T23:55:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.030+0000 7f4944c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4940102820 0x7f4940199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:34558/0 (socket says 172.21.15.12:34558) 2024-03-17T23:55:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.030+0000 7f4945600640 1 -- 172.21.15.12:0/2634165424 learned_addr learned my addr 172.21.15.12:0/2634165424 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.031+0000 7f4946000640 1 -- 172.21.15.12:0/2634165424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4940102820 msgr2=0x7f4940199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.031+0000 7f4946000640 1 --2- 172.21.15.12:0/2634165424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4940102820 0x7f4940199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.031+0000 7f4946000640 1 -- 172.21.15.12:0/2634165424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4940101630 msgr2=0x7f4940199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.031+0000 7f4946000640 1 --2- 172.21.15.12:0/2634165424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4940101630 0x7f4940199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.031+0000 7f4946000640 1 -- 172.21.15.12:0/2634165424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49401a16d0 con 0x7f49401031e0 2024-03-17T23:55:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.031+0000 7f4946000640 1 --2- 172.21.15.12:0/2634165424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49401031e0 0x7f49401a1060 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f493800ec50 tx=0x7f493800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.031+0000 7f493e000640 1 -- 172.21.15.12:0/2634165424 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4938003b70 con 0x7f49401031e0 2024-03-17T23:55:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.031+0000 7f493e000640 1 -- 172.21.15.12:0/2634165424 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4938003d10 con 0x7f49401031e0 2024-03-17T23:55:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.031+0000 7f4946a00640 1 -- 172.21.15.12:0/2634165424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49401a19c0 con 0x7f49401031e0 2024-03-17T23:55:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.031+0000 7f493e000640 1 -- 172.21.15.12:0/2634165424 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4938010890 con 0x7f49401031e0 2024-03-17T23:55:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.031+0000 7f4946a00640 1 -- 172.21.15.12:0/2634165424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49401a1e90 con 0x7f49401031e0 2024-03-17T23:55:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.033+0000 7f4946a00640 1 -- 172.21.15.12:0/2634165424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49401a20c0 con 0x7f49401031e0 2024-03-17T23:55:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.036+0000 7f493e000640 1 -- 172.21.15.12:0/2634165424 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4938003fc0 con 0x7f49401031e0 2024-03-17T23:55:30.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.037+0000 7f493e000640 1 --2- 172.21.15.12:0/2634165424 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f49280a2890 0x7f49280a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:30.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.038+0000 7f493e000640 1 -- 172.21.15.12:0/2634165424 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f49380cf410 con 0x7f49401031e0 2024-03-17T23:55:30.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.038+0000 7f4945600640 1 --2- 172.21.15.12:0/2634165424 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f49280a2890 0x7f49280a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:30.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.038+0000 7f4945600640 1 --2- 172.21.15.12:0/2634165424 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f49280a2890 0x7f49280a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f49340058c0 tx=0x7f4934005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:30.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.046+0000 7f493e000640 1 -- 172.21.15.12:0/2634165424 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f493809c0b0 con 0x7f49401031e0 2024-03-17T23:55:30.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:30 smithi012 ceph-mon[31541]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-17T23:55:30.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:30 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2158995707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:55:30.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:30 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3768899117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:30.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.246+0000 7f4946a00640 1 -- 172.21.15.12:0/2634165424 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4940075130 con 0x7f49280a2890 2024-03-17T23:55:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.262+0000 7f493e000640 1 -- 172.21.15.12:0/2634165424 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340718 (secure 0 0 0) 0x7f4940075130 con 0x7f49280a2890 2024-03-17T23:55:30.268 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:30.271 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-17T23:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.270+0000 7f4946a00640 1 -- 172.21.15.12:0/2634165424 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f49280a2890 msgr2=0x7f49280a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.270+0000 7f4946a00640 1 --2- 172.21.15.12:0/2634165424 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f49280a2890 0x7f49280a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f49340058c0 tx=0x7f4934005f90 comp rx=0 tx=0).stop 2024-03-17T23:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.271+0000 7f4946a00640 1 -- 172.21.15.12:0/2634165424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49401031e0 msgr2=0x7f49401a1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.271+0000 7f4946a00640 1 --2- 172.21.15.12:0/2634165424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49401031e0 0x7f49401a1060 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f493800ec50 tx=0x7f493800c6a0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.271+0000 7f4946a00640 1 -- 172.21.15.12:0/2634165424 shutdown_connections 2024-03-17T23:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.271+0000 7f4946a00640 1 --2- 172.21.15.12:0/2634165424 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f49280a2890 0x7f49280a4d50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.271+0000 7f4946a00640 1 --2- 172.21.15.12:0/2634165424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49401031e0 0x7f49401a1060 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.271+0000 7f4946a00640 1 --2- 172.21.15.12:0/2634165424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4940102820 0x7f4940199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.271+0000 7f4946a00640 1 --2- 172.21.15.12:0/2634165424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4940101630 0x7f4940199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.271+0000 7f4946a00640 1 -- 172.21.15.12:0/2634165424 >> 172.21.15.12:0/2634165424 conn(0x7f49400fcbc0 msgr2=0x7f4940073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:30.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.271+0000 7f4946a00640 1 -- 172.21.15.12:0/2634165424 shutdown_connections 2024-03-17T23:55:30.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.271+0000 7f4946a00640 1 -- 172.21.15.12:0/2634165424 wait complete. 2024-03-17T23:55:30.291 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-03-17T23:55:30.108634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601167,"num_objects":97,"num_object_clones":0,"num_object_copies":293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":154,"num_read_kb":133,"num_write":384,"num_write_kb":3009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":316,"ondisk_log_size":316,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":352872,"kb_used_data":12040,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588952,"statfs":{"total":1151940427776,"available":1151579086848,"internally_reserved":0,"allocated":12328960,"data_stored":7736513,"data_compressed":286993,"data_compressed_allocated":389120,"data_compressed_original":2277376,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6300,"num_objects":73,"num_object_clones":0,"num_object_copies":221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":155,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006315"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660413+0000","last_change":"2024-03-17T23:54:56.354062+0000","last_active":"2024-03-17T23:55:04.660413+0000","last_peered":"2024-03-17T23:55:04.660413+0000","last_clean":"2024-03-17T23:55:04.660413+0000","last_became_active":"2024-03-17T23:54:56.353753+0000","last_became_peered":"2024-03-17T23:54:56.353753+0000","last_unstale":"2024-03-17T23:55:04.660413+0000","last_undegraded":"2024-03-17T23:55:04.660413+0000","last_fullsized":"2024-03-17T23:55:04.660413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:41:50.385420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.663922+0000","last_change":"2024-03-17T23:54:56.356507+0000","last_active":"2024-03-17T23:55:04.663922+0000","last_peered":"2024-03-17T23:55:04.663922+0000","last_clean":"2024-03-17T23:55:04.663922+0000","last_became_active":"2024-03-17T23:54:56.355714+0000","last_became_peered":"2024-03-17T23:54:56.355714+0000","last_unstale":"2024-03-17T23:55:04.663922+0000","last_undegraded":"2024-03-17T23:55:04.663922+0000","last_fullsized":"2024-03-17T23:55:04.663922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:44:15.992630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660980+0000","last_change":"2024-03-17T23:54:56.352207+0000","last_active":"2024-03-17T23:55:04.660980+0000","last_peered":"2024-03-17T23:55:04.660980+0000","last_clean":"2024-03-17T23:55:04.660980+0000","last_became_active":"2024-03-17T23:54:56.351704+0000","last_became_peered":"2024-03-17T23:54:56.351704+0000","last_unstale":"2024-03-17T23:55:04.660980+0000","last_undegraded":"2024-03-17T23:55:04.660980+0000","last_fullsized":"2024-03-17T23:55:04.660980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:34.328120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222439+0000","last_change":"2024-03-17T23:54:56.343336+0000","last_active":"2024-03-17T23:55:03.222439+0000","last_peered":"2024-03-17T23:55:03.222439+0000","last_clean":"2024-03-17T23:55:03.222439+0000","last_became_active":"2024-03-17T23:54:56.343033+0000","last_became_peered":"2024-03-17T23:54:56.343033+0000","last_unstale":"2024-03-17T23:55:03.222439+0000","last_undegraded":"2024-03-17T23:55:03.222439+0000","last_fullsized":"2024-03-17T23:55:03.222439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:40:21.814558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.178101+0000","last_change":"2024-03-17T23:54:56.360541+0000","last_active":"2024-03-17T23:55:03.178101+0000","last_peered":"2024-03-17T23:55:03.178101+0000","last_clean":"2024-03-17T23:55:03.178101+0000","last_became_active":"2024-03-17T23:54:56.360204+0000","last_became_peered":"2024-03-17T23:54:56.360204+0000","last_unstale":"2024-03-17T23:55:03.178101+0000","last_undegraded":"2024-03-17T23:55:03.178101+0000","last_fullsized":"2024-03-17T23:55:03.178101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:57:37.853168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659365+0000","last_change":"2024-03-17T23:54:56.363727+0000","last_active":"2024-03-17T23:55:04.659365+0000","last_peered":"2024-03-17T23:55:04.659365+0000","last_clean":"2024-03-17T23:55:04.659365+0000","last_became_active":"2024-03-17T23:54:56.363399+0000","last_became_peered":"2024-03-17T23:54:56.363399+0000","last_unstale":"2024-03-17T23:55:04.659365+0000","last_undegraded":"2024-03-17T23:55:04.659365+0000","last_fullsized":"2024-03-17T23:55:04.659365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:42.451379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306609+0000","last_change":"2024-03-17T23:54:56.361881+0000","last_active":"2024-03-17T23:55:03.306609+0000","last_peered":"2024-03-17T23:55:03.306609+0000","last_clean":"2024-03-17T23:55:03.306609+0000","last_became_active":"2024-03-17T23:54:56.361531+0000","last_became_peered":"2024-03-17T23:54:56.361531+0000","last_unstale":"2024-03-17T23:55:03.306609+0000","last_undegraded":"2024-03-17T23:55:03.306609+0000","last_fullsized":"2024-03-17T23:55:03.306609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:44:39.792763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.371774+0000","last_change":"2024-03-17T23:54:56.358429+0000","last_active":"2024-03-17T23:55:03.371774+0000","last_peered":"2024-03-17T23:55:03.371774+0000","last_clean":"2024-03-17T23:55:03.371774+0000","last_became_active":"2024-03-17T23:54:56.355268+0000","last_became_peered":"2024-03-17T23:54:56.355268+0000","last_unstale":"2024-03-17T23:55:03.371774+0000","last_undegraded":"2024-03-17T23:55:03.371774+0000","last_fullsized":"2024-03-17T23:55:03.371774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:00:37.687542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853925+0000","last_change":"2024-03-17T23:54:56.344228+0000","last_active":"2024-03-17T23:55:02.853925+0000","last_peered":"2024-03-17T23:55:02.853925+0000","last_clean":"2024-03-17T23:55:02.853925+0000","last_became_active":"2024-03-17T23:54:56.343863+0000","last_became_peered":"2024-03-17T23:54:56.343863+0000","last_unstale":"2024-03-17T23:55:02.853925+0000","last_undegraded":"2024-03-17T23:55:02.853925+0000","last_fullsized":"2024-03-17T23:55:02.853925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:45:12.552611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659558+0000","last_change":"2024-03-17T23:54:56.341215+0000","last_active":"2024-03-17T23:55:04.659558+0000","last_peered":"2024-03-17T23:55:04.659558+0000","last_clean":"2024-03-17T23:55:04.659558+0000","last_became_active":"2024-03-17T23:54:56.340850+0000","last_became_peered":"2024-03-17T23:54:56.340850+0000","last_unstale":"2024-03-17T23:55:04.659558+0000","last_undegraded":"2024-03-17T23:55:04.659558+0000","last_fullsized":"2024-03-17T23:55:04.659558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:28.273320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306995+0000","last_change":"2024-03-17T23:54:56.346613+0000","last_active":"2024-03-17T23:55:03.306995+0000","last_peered":"2024-03-17T23:55:03.306995+0000","last_clean":"2024-03-17T23:55:03.306995+0000","last_became_active":"2024-03-17T23:54:56.345907+0000","last_became_peered":"2024-03-17T23:54:56.345907+0000","last_unstale":"2024-03-17T23:55:03.306995+0000","last_undegraded":"2024-03-17T23:55:03.306995+0000","last_fullsized":"2024-03-17T23:55:03.306995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:21.493841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222384+0000","last_change":"2024-03-17T23:54:56.341924+0000","last_active":"2024-03-17T23:55:03.222384+0000","last_peered":"2024-03-17T23:55:03.222384+0000","last_clean":"2024-03-17T23:55:03.222384+0000","last_became_active":"2024-03-17T23:54:56.341469+0000","last_became_peered":"2024-03-17T23:54:56.341469+0000","last_unstale":"2024-03-17T23:55:03.222384+0000","last_undegraded":"2024-03-17T23:55:03.222384+0000","last_fullsized":"2024-03-17T23:55:03.222384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:49:47.256954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.858501+0000","last_change":"2024-03-17T23:54:56.332469+0000","last_active":"2024-03-17T23:55:02.858501+0000","last_peered":"2024-03-17T23:55:02.858501+0000","last_clean":"2024-03-17T23:55:02.858501+0000","last_became_active":"2024-03-17T23:54:56.330684+0000","last_became_peered":"2024-03-17T23:54:56.330684+0000","last_unstale":"2024-03-17T23:55:02.858501+0000","last_undegraded":"2024-03-17T23:55:02.858501+0000","last_fullsized":"2024-03-17T23:55:02.858501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:25.211711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660643+0000","last_change":"2024-03-17T23:54:56.354451+0000","last_active":"2024-03-17T23:55:04.660643+0000","last_peered":"2024-03-17T23:55:04.660643+0000","last_clean":"2024-03-17T23:55:04.660643+0000","last_became_active":"2024-03-17T23:54:56.354160+0000","last_became_peered":"2024-03-17T23:54:56.354160+0000","last_unstale":"2024-03-17T23:55:04.660643+0000","last_undegraded":"2024-03-17T23:55:04.660643+0000","last_fullsized":"2024-03-17T23:55:04.660643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:07:37.644444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660653+0000","last_change":"2024-03-17T23:54:56.353705+0000","last_active":"2024-03-17T23:55:04.660653+0000","last_peered":"2024-03-17T23:55:04.660653+0000","last_clean":"2024-03-17T23:55:04.660653+0000","last_became_active":"2024-03-17T23:54:56.352389+0000","last_became_peered":"2024-03-17T23:54:56.352389+0000","last_unstale":"2024-03-17T23:55:04.660653+0000","last_undegraded":"2024-03-17T23:55:04.660653+0000","last_fullsized":"2024-03-17T23:55:04.660653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:17.153753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853481+0000","last_change":"2024-03-17T23:54:56.348467+0000","last_active":"2024-03-17T23:55:02.853481+0000","last_peered":"2024-03-17T23:55:02.853481+0000","last_clean":"2024-03-17T23:55:02.853481+0000","last_became_active":"2024-03-17T23:54:56.348118+0000","last_became_peered":"2024-03-17T23:54:56.348118+0000","last_unstale":"2024-03-17T23:55:02.853481+0000","last_undegraded":"2024-03-17T23:55:02.853481+0000","last_fullsized":"2024-03-17T23:55:02.853481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:53:45.717303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660631+0000","last_change":"2024-03-17T23:54:56.354117+0000","last_active":"2024-03-17T23:55:04.660631+0000","last_peered":"2024-03-17T23:55:04.660631+0000","last_clean":"2024-03-17T23:55:04.660631+0000","last_became_active":"2024-03-17T23:54:56.353818+0000","last_became_peered":"2024-03-17T23:54:56.353818+0000","last_unstale":"2024-03-17T23:55:04.660631+0000","last_undegraded":"2024-03-17T23:55:04.660631+0000","last_fullsized":"2024-03-17T23:55:04.660631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:09:17.314024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.449909+0000","last_change":"2024-03-17T23:54:56.363781+0000","last_active":"2024-03-17T23:55:04.449909+0000","last_peered":"2024-03-17T23:55:04.449909+0000","last_clean":"2024-03-17T23:55:04.449909+0000","last_became_active":"2024-03-17T23:54:56.363364+0000","last_became_peered":"2024-03-17T23:54:56.363364+0000","last_unstale":"2024-03-17T23:55:04.449909+0000","last_undegraded":"2024-03-17T23:55:04.449909+0000","last_fullsized":"2024-03-17T23:55:04.449909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:21:00.395329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207852+0000","last_change":"2024-03-17T23:54:56.360905+0000","last_active":"2024-03-17T23:55:04.207852+0000","last_peered":"2024-03-17T23:55:04.207852+0000","last_clean":"2024-03-17T23:55:04.207852+0000","last_became_active":"2024-03-17T23:54:56.360492+0000","last_became_peered":"2024-03-17T23:54:56.360492+0000","last_unstale":"2024-03-17T23:55:04.207852+0000","last_undegraded":"2024-03-17T23:55:04.207852+0000","last_fullsized":"2024-03-17T23:55:04.207852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:34:42.440635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207652+0000","last_change":"2024-03-17T23:55:01.889086+0000","last_active":"2024-03-17T23:55:04.207652+0000","last_peered":"2024-03-17T23:55:04.207652+0000","last_clean":"2024-03-17T23:55:04.207652+0000","last_became_active":"2024-03-17T23:55:01.888505+0000","last_became_peered":"2024-03-17T23:55:01.888505+0000","last_unstale":"2024-03-17T23:55:04.207652+0000","last_undegraded":"2024-03-17T23:55:04.207652+0000","last_fullsized":"2024-03-17T23:55:04.207652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:00:43.490360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.042292+0000","last_change":"2024-03-17T23:54:55.858790+0000","last_active":"2024-03-17T23:55:28.042292+0000","last_peered":"2024-03-17T23:55:28.042292+0000","last_clean":"2024-03-17T23:55:28.042292+0000","last_became_active":"2024-03-17T23:54:55.858437+0000","last_became_peered":"2024-03-17T23:54:55.858437+0000","last_unstale":"2024-03-17T23:55:28.042292+0000","last_undegraded":"2024-03-17T23:55:28.042292+0000","last_fullsized":"2024-03-17T23:55:28.042292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:49:26.830471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.372883+0000","last_change":"2024-03-17T23:54:56.356641+0000","last_active":"2024-03-17T23:55:03.372883+0000","last_peered":"2024-03-17T23:55:03.372883+0000","last_clean":"2024-03-17T23:55:03.372883+0000","last_became_active":"2024-03-17T23:54:56.355402+0000","last_became_peered":"2024-03-17T23:54:56.355402+0000","last_unstale":"2024-03-17T23:55:03.372883+0000","last_undegraded":"2024-03-17T23:55:03.372883+0000","last_fullsized":"2024-03-17T23:55:03.372883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:01:04.699357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222348+0000","last_change":"2024-03-17T23:55:01.888632+0000","last_active":"2024-03-17T23:55:03.222348+0000","last_peered":"2024-03-17T23:55:03.222348+0000","last_clean":"2024-03-17T23:55:03.222348+0000","last_became_active":"2024-03-17T23:55:01.888240+0000","last_became_peered":"2024-03-17T23:55:01.888240+0000","last_unstale":"2024-03-17T23:55:03.222348+0000","last_undegraded":"2024-03-17T23:55:03.222348+0000","last_fullsized":"2024-03-17T23:55:03.222348+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:02:04.562112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.372960+0000","last_change":"2024-03-17T23:54:55.871842+0000","last_active":"2024-03-17T23:55:03.372960+0000","last_peered":"2024-03-17T23:55:03.372960+0000","last_clean":"2024-03-17T23:55:03.372960+0000","last_became_active":"2024-03-17T23:54:55.870965+0000","last_became_peered":"2024-03-17T23:54:55.870965+0000","last_unstale":"2024-03-17T23:55:03.372960+0000","last_undegraded":"2024-03-17T23:55:03.372960+0000","last_fullsized":"2024-03-17T23:55:03.372960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:27:04.762902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.661453+0000","last_change":"2024-03-17T23:54:56.368277+0000","last_active":"2024-03-17T23:55:04.661453+0000","last_peered":"2024-03-17T23:55:04.661453+0000","last_clean":"2024-03-17T23:55:04.661453+0000","last_became_active":"2024-03-17T23:54:56.366229+0000","last_became_peered":"2024-03-17T23:54:56.366229+0000","last_unstale":"2024-03-17T23:55:04.661453+0000","last_undegraded":"2024-03-17T23:55:04.661453+0000","last_fullsized":"2024-03-17T23:55:04.661453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:50:05.241745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222480+0000","last_change":"2024-03-17T23:55:01.881731+0000","last_active":"2024-03-17T23:55:03.222480+0000","last_peered":"2024-03-17T23:55:03.222480+0000","last_clean":"2024-03-17T23:55:03.222480+0000","last_became_active":"2024-03-17T23:55:01.881383+0000","last_became_peered":"2024-03-17T23:55:01.881383+0000","last_unstale":"2024-03-17T23:55:03.222480+0000","last_undegraded":"2024-03-17T23:55:03.222480+0000","last_fullsized":"2024-03-17T23:55:03.222480+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:45.095672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.028239+0000","last_change":"2024-03-17T23:54:55.838014+0000","last_active":"2024-03-17T23:55:28.028239+0000","last_peered":"2024-03-17T23:55:28.028239+0000","last_clean":"2024-03-17T23:55:28.028239+0000","last_became_active":"2024-03-17T23:54:55.837694+0000","last_became_peered":"2024-03-17T23:54:55.837694+0000","last_unstale":"2024-03-17T23:55:28.028239+0000","last_undegraded":"2024-03-17T23:55:28.028239+0000","last_fullsized":"2024-03-17T23:55:28.028239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:16:01.173460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853699+0000","last_change":"2024-03-17T23:54:56.343240+0000","last_active":"2024-03-17T23:55:02.853699+0000","last_peered":"2024-03-17T23:55:02.853699+0000","last_clean":"2024-03-17T23:55:02.853699+0000","last_became_active":"2024-03-17T23:54:56.342928+0000","last_became_peered":"2024-03-17T23:54:56.342928+0000","last_unstale":"2024-03-17T23:55:02.853699+0000","last_undegraded":"2024-03-17T23:55:02.853699+0000","last_fullsized":"2024-03-17T23:55:02.853699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:33:43.922886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222266+0000","last_change":"2024-03-17T23:55:01.880207+0000","last_active":"2024-03-17T23:55:03.222266+0000","last_peered":"2024-03-17T23:55:03.222266+0000","last_clean":"2024-03-17T23:55:03.222266+0000","last_became_active":"2024-03-17T23:55:01.879748+0000","last_became_peered":"2024-03-17T23:55:01.879748+0000","last_unstale":"2024-03-17T23:55:03.222266+0000","last_undegraded":"2024-03-17T23:55:03.222266+0000","last_fullsized":"2024-03-17T23:55:03.222266+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:33.782782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.032972+0000","last_change":"2024-03-17T23:54:55.839822+0000","last_active":"2024-03-17T23:55:27.032972+0000","last_peered":"2024-03-17T23:55:27.032972+0000","last_clean":"2024-03-17T23:55:27.032972+0000","last_became_active":"2024-03-17T23:54:55.839506+0000","last_became_peered":"2024-03-17T23:54:55.839506+0000","last_unstale":"2024-03-17T23:55:27.032972+0000","last_undegraded":"2024-03-17T23:55:27.032972+0000","last_fullsized":"2024-03-17T23:55:27.032972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:08.743582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854388+0000","last_change":"2024-03-17T23:54:56.346914+0000","last_active":"2024-03-17T23:55:02.854388+0000","last_peered":"2024-03-17T23:55:02.854388+0000","last_clean":"2024-03-17T23:55:02.854388+0000","last_became_active":"2024-03-17T23:54:56.346620+0000","last_became_peered":"2024-03-17T23:54:56.346620+0000","last_unstale":"2024-03-17T23:55:02.854388+0000","last_undegraded":"2024-03-17T23:55:02.854388+0000","last_fullsized":"2024-03-17T23:55:02.854388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:44:58.643614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207738+0000","last_change":"2024-03-17T23:55:01.864527+0000","last_active":"2024-03-17T23:55:04.207738+0000","last_peered":"2024-03-17T23:55:04.207738+0000","last_clean":"2024-03-17T23:55:04.207738+0000","last_became_active":"2024-03-17T23:55:01.864106+0000","last_became_peered":"2024-03-17T23:55:01.864106+0000","last_unstale":"2024-03-17T23:55:04.207738+0000","last_undegraded":"2024-03-17T23:55:04.207738+0000","last_fullsized":"2024-03-17T23:55:04.207738+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:27:19.811639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:23.988262+0000","last_change":"2024-03-17T23:54:55.842156+0000","last_active":"2024-03-17T23:55:23.988262+0000","last_peered":"2024-03-17T23:55:23.988262+0000","last_clean":"2024-03-17T23:55:23.988262+0000","last_became_active":"2024-03-17T23:54:55.841858+0000","last_became_peered":"2024-03-17T23:54:55.841858+0000","last_unstale":"2024-03-17T23:55:23.988262+0000","last_undegraded":"2024-03-17T23:55:23.988262+0000","last_fullsized":"2024-03-17T23:55:23.988262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:01:05.967638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.178220+0000","last_change":"2024-03-17T23:54:56.357770+0000","last_active":"2024-03-17T23:55:03.178220+0000","last_peered":"2024-03-17T23:55:03.178220+0000","last_clean":"2024-03-17T23:55:03.178220+0000","last_became_active":"2024-03-17T23:54:56.357411+0000","last_became_peered":"2024-03-17T23:54:56.357411+0000","last_unstale":"2024-03-17T23:55:03.178220+0000","last_undegraded":"2024-03-17T23:55:03.178220+0000","last_fullsized":"2024-03-17T23:55:03.178220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:01:04.685138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207636+0000","last_change":"2024-03-17T23:55:01.883104+0000","last_active":"2024-03-17T23:55:04.207636+0000","last_peered":"2024-03-17T23:55:04.207636+0000","last_clean":"2024-03-17T23:55:04.207636+0000","last_became_active":"2024-03-17T23:55:01.882650+0000","last_became_peered":"2024-03-17T23:55:01.882650+0000","last_unstale":"2024-03-17T23:55:04.207636+0000","last_undegraded":"2024-03-17T23:55:04.207636+0000","last_fullsized":"2024-03-17T23:55:04.207636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:54:18.420997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:25.000312+0000","last_change":"2024-03-17T23:54:55.854625+0000","last_active":"2024-03-17T23:55:25.000312+0000","last_peered":"2024-03-17T23:55:25.000312+0000","last_clean":"2024-03-17T23:55:25.000312+0000","last_became_active":"2024-03-17T23:54:55.854185+0000","last_became_peered":"2024-03-17T23:54:55.854185+0000","last_unstale":"2024-03-17T23:55:25.000312+0000","last_undegraded":"2024-03-17T23:55:25.000312+0000","last_fullsized":"2024-03-17T23:55:25.000312+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:42:13.424550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854342+0000","last_change":"2024-03-17T23:54:56.353311+0000","last_active":"2024-03-17T23:55:02.854342+0000","last_peered":"2024-03-17T23:55:02.854342+0000","last_clean":"2024-03-17T23:55:02.854342+0000","last_became_active":"2024-03-17T23:54:56.353013+0000","last_became_peered":"2024-03-17T23:54:56.353013+0000","last_unstale":"2024-03-17T23:55:02.854342+0000","last_undegraded":"2024-03-17T23:55:02.854342+0000","last_fullsized":"2024-03-17T23:55:02.854342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:21:20.119186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660812+0000","last_change":"2024-03-17T23:55:01.888595+0000","last_active":"2024-03-17T23:55:04.660812+0000","last_peered":"2024-03-17T23:55:04.660812+0000","last_clean":"2024-03-17T23:55:04.660812+0000","last_became_active":"2024-03-17T23:55:01.888154+0000","last_became_peered":"2024-03-17T23:55:01.888154+0000","last_unstale":"2024-03-17T23:55:04.660812+0000","last_undegraded":"2024-03-17T23:55:04.660812+0000","last_fullsized":"2024-03-17T23:55:04.660812+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:28:17.680458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:25.003941+0000","last_change":"2024-03-17T23:54:55.862756+0000","last_active":"2024-03-17T23:55:25.003941+0000","last_peered":"2024-03-17T23:55:25.003941+0000","last_clean":"2024-03-17T23:55:25.003941+0000","last_became_active":"2024-03-17T23:54:55.862298+0000","last_became_peered":"2024-03-17T23:54:55.862298+0000","last_unstale":"2024-03-17T23:55:25.003941+0000","last_undegraded":"2024-03-17T23:55:25.003941+0000","last_fullsized":"2024-03-17T23:55:25.003941+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:15:52.613136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.371626+0000","last_change":"2024-03-17T23:54:56.354359+0000","last_active":"2024-03-17T23:55:03.371626+0000","last_peered":"2024-03-17T23:55:03.371626+0000","last_clean":"2024-03-17T23:55:03.371626+0000","last_became_active":"2024-03-17T23:54:56.354002+0000","last_became_peered":"2024-03-17T23:54:56.354002+0000","last_unstale":"2024-03-17T23:55:03.371626+0000","last_undegraded":"2024-03-17T23:55:03.371626+0000","last_fullsized":"2024-03-17T23:55:03.371626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:41:12.128471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.664012+0000","last_change":"2024-03-17T23:55:01.880579+0000","last_active":"2024-03-17T23:55:04.664012+0000","last_peered":"2024-03-17T23:55:04.664012+0000","last_clean":"2024-03-17T23:55:04.664012+0000","last_became_active":"2024-03-17T23:55:01.879471+0000","last_became_peered":"2024-03-17T23:55:01.879471+0000","last_unstale":"2024-03-17T23:55:04.664012+0000","last_undegraded":"2024-03-17T23:55:04.664012+0000","last_fullsized":"2024-03-17T23:55:04.664012+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:51.117904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:26.011593+0000","last_change":"2024-03-17T23:54:55.842998+0000","last_active":"2024-03-17T23:55:26.011593+0000","last_peered":"2024-03-17T23:55:26.011593+0000","last_clean":"2024-03-17T23:55:26.011593+0000","last_became_active":"2024-03-17T23:54:55.842678+0000","last_became_peered":"2024-03-17T23:54:55.842678+0000","last_unstale":"2024-03-17T23:55:26.011593+0000","last_undegraded":"2024-03-17T23:55:26.011593+0000","last_fullsized":"2024-03-17T23:55:26.011593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:14.546738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.449645+0000","last_change":"2024-03-17T23:54:56.365881+0000","last_active":"2024-03-17T23:55:04.449645+0000","last_peered":"2024-03-17T23:55:04.449645+0000","last_clean":"2024-03-17T23:55:04.449645+0000","last_became_active":"2024-03-17T23:54:56.365560+0000","last_became_peered":"2024-03-17T23:54:56.365560+0000","last_unstale":"2024-03-17T23:55:04.449645+0000","last_undegraded":"2024-03-17T23:55:04.449645+0000","last_fullsized":"2024-03-17T23:55:04.449645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:38.436644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.371883+0000","last_change":"2024-03-17T23:55:01.882651+0000","last_active":"2024-03-17T23:55:03.371883+0000","last_peered":"2024-03-17T23:55:03.371883+0000","last_clean":"2024-03-17T23:55:03.371883+0000","last_became_active":"2024-03-17T23:55:01.881717+0000","last_became_peered":"2024-03-17T23:55:01.881717+0000","last_unstale":"2024-03-17T23:55:03.371883+0000","last_undegraded":"2024-03-17T23:55:03.371883+0000","last_fullsized":"2024-03-17T23:55:03.371883+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:21:08.644184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:23.987241+0000","last_change":"2024-03-17T23:54:55.854822+0000","last_active":"2024-03-17T23:55:23.987241+0000","last_peered":"2024-03-17T23:55:23.987241+0000","last_clean":"2024-03-17T23:55:23.987241+0000","last_became_active":"2024-03-17T23:54:55.854501+0000","last_became_peered":"2024-03-17T23:54:55.854501+0000","last_unstale":"2024-03-17T23:55:23.987241+0000","last_undegraded":"2024-03-17T23:55:23.987241+0000","last_fullsized":"2024-03-17T23:55:23.987241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:04:12.504798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853839+0000","last_change":"2024-03-17T23:54:56.361898+0000","last_active":"2024-03-17T23:55:02.853839+0000","last_peered":"2024-03-17T23:55:02.853839+0000","last_clean":"2024-03-17T23:55:02.853839+0000","last_became_active":"2024-03-17T23:54:56.361556+0000","last_became_peered":"2024-03-17T23:54:56.361556+0000","last_unstale":"2024-03-17T23:55:02.853839+0000","last_undegraded":"2024-03-17T23:55:02.853839+0000","last_fullsized":"2024-03-17T23:55:02.853839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:52:08.932801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.450498+0000","last_change":"2024-03-17T23:55:01.882628+0000","last_active":"2024-03-17T23:55:04.450498+0000","last_peered":"2024-03-17T23:55:04.450498+0000","last_clean":"2024-03-17T23:55:04.450498+0000","last_became_active":"2024-03-17T23:55:01.881886+0000","last_became_peered":"2024-03-17T23:55:01.881886+0000","last_unstale":"2024-03-17T23:55:04.450498+0000","last_undegraded":"2024-03-17T23:55:04.450498+0000","last_fullsized":"2024-03-17T23:55:04.450498+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:22:58.534442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.032189+0000","last_change":"2024-03-17T23:54:55.839925+0000","last_active":"2024-03-17T23:55:27.032189+0000","last_peered":"2024-03-17T23:55:27.032189+0000","last_clean":"2024-03-17T23:55:27.032189+0000","last_became_active":"2024-03-17T23:54:55.839626+0000","last_became_peered":"2024-03-17T23:54:55.839626+0000","last_unstale":"2024-03-17T23:55:27.032189+0000","last_undegraded":"2024-03-17T23:55:27.032189+0000","last_fullsized":"2024-03-17T23:55:27.032189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:39.456462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.661735+0000","last_change":"2024-03-17T23:54:56.352766+0000","last_active":"2024-03-17T23:55:04.661735+0000","last_peered":"2024-03-17T23:55:04.661735+0000","last_clean":"2024-03-17T23:55:04.661735+0000","last_became_active":"2024-03-17T23:54:56.351817+0000","last_became_peered":"2024-03-17T23:54:56.351817+0000","last_unstale":"2024-03-17T23:55:04.661735+0000","last_undegraded":"2024-03-17T23:55:04.661735+0000","last_fullsized":"2024-03-17T23:55:04.661735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:57:38.097163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222388+0000","last_change":"2024-03-17T23:55:01.892650+0000","last_active":"2024-03-17T23:55:03.222388+0000","last_peered":"2024-03-17T23:55:03.222388+0000","last_clean":"2024-03-17T23:55:03.222388+0000","last_became_active":"2024-03-17T23:55:01.892269+0000","last_became_peered":"2024-03-17T23:55:01.892269+0000","last_unstale":"2024-03-17T23:55:03.222388+0000","last_undegraded":"2024-03-17T23:55:03.222388+0000","last_fullsized":"2024-03-17T23:55:03.222388+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:36:17.525339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:23.988745+0000","last_change":"2024-03-17T23:54:55.863264+0000","last_active":"2024-03-17T23:55:23.988745+0000","last_peered":"2024-03-17T23:55:23.988745+0000","last_clean":"2024-03-17T23:55:23.988745+0000","last_became_active":"2024-03-17T23:54:55.863000+0000","last_became_peered":"2024-03-17T23:54:55.863000+0000","last_unstale":"2024-03-17T23:55:23.988745+0000","last_undegraded":"2024-03-17T23:55:23.988745+0000","last_fullsized":"2024-03-17T23:55:23.988745+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:58.208064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853795+0000","last_change":"2024-03-17T23:54:56.351161+0000","last_active":"2024-03-17T23:55:02.853795+0000","last_peered":"2024-03-17T23:55:02.853795+0000","last_clean":"2024-03-17T23:55:02.853795+0000","last_became_active":"2024-03-17T23:54:56.349787+0000","last_became_peered":"2024-03-17T23:54:56.349787+0000","last_unstale":"2024-03-17T23:55:02.853795+0000","last_undegraded":"2024-03-17T23:55:02.853795+0000","last_fullsized":"2024-03-17T23:55:02.853795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:19:28.635060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.451325+0000","last_change":"2024-03-17T23:55:01.882229+0000","last_active":"2024-03-17T23:55:04.451325+0000","last_peered":"2024-03-17T23:55:04.451325+0000","last_clean":"2024-03-17T23:55:04.451325+0000","last_became_active":"2024-03-17T23:55:01.881768+0000","last_became_peered":"2024-03-17T23:55:01.881768+0000","last_unstale":"2024-03-17T23:55:04.451325+0000","last_undegraded":"2024-03-17T23:55:04.451325+0000","last_fullsized":"2024-03-17T23:55:04.451325+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:03:35.149413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'12","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:23.986986+0000","last_change":"2024-03-17T23:54:55.846449+0000","last_active":"2024-03-17T23:55:23.986986+0000","last_peered":"2024-03-17T23:55:23.986986+0000","last_clean":"2024-03-17T23:55:23.986986+0000","last_became_active":"2024-03-17T23:54:55.846064+0000","last_became_peered":"2024-03-17T23:54:55.846064+0000","last_unstale":"2024-03-17T23:55:23.986986+0000","last_undegraded":"2024-03-17T23:55:23.986986+0000","last_fullsized":"2024-03-17T23:55:23.986986+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:45.985724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.178254+0000","last_change":"2024-03-17T23:54:56.343717+0000","last_active":"2024-03-17T23:55:03.178254+0000","last_peered":"2024-03-17T23:55:03.178254+0000","last_clean":"2024-03-17T23:55:03.178254+0000","last_became_active":"2024-03-17T23:54:56.343418+0000","last_became_peered":"2024-03-17T23:54:56.343418+0000","last_unstale":"2024-03-17T23:55:03.178254+0000","last_undegraded":"2024-03-17T23:55:03.178254+0000","last_fullsized":"2024-03-17T23:55:03.178254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:11:40.402878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854876+0000","last_change":"2024-03-17T23:55:01.884671+0000","last_active":"2024-03-17T23:55:02.854876+0000","last_peered":"2024-03-17T23:55:02.854876+0000","last_clean":"2024-03-17T23:55:02.854876+0000","last_became_active":"2024-03-17T23:55:01.884102+0000","last_became_peered":"2024-03-17T23:55:01.884102+0000","last_unstale":"2024-03-17T23:55:02.854876+0000","last_undegraded":"2024-03-17T23:55:02.854876+0000","last_fullsized":"2024-03-17T23:55:02.854876+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:53:31.198466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.039040+0000","last_change":"2024-03-17T23:54:55.862124+0000","last_active":"2024-03-17T23:55:28.039040+0000","last_peered":"2024-03-17T23:55:28.039040+0000","last_clean":"2024-03-17T23:55:28.039040+0000","last_became_active":"2024-03-17T23:54:55.861758+0000","last_became_peered":"2024-03-17T23:54:55.861758+0000","last_unstale":"2024-03-17T23:55:28.039040+0000","last_undegraded":"2024-03-17T23:55:28.039040+0000","last_fullsized":"2024-03-17T23:55:28.039040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:02:25.138388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.858015+0000","last_change":"2024-03-17T23:54:56.361699+0000","last_active":"2024-03-17T23:55:02.858015+0000","last_peered":"2024-03-17T23:55:02.858015+0000","last_clean":"2024-03-17T23:55:02.858015+0000","last_became_active":"2024-03-17T23:54:56.361365+0000","last_became_peered":"2024-03-17T23:54:56.361365+0000","last_unstale":"2024-03-17T23:55:02.858015+0000","last_undegraded":"2024-03-17T23:55:02.858015+0000","last_fullsized":"2024-03-17T23:55:02.858015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:24:30.649956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660140+0000","last_change":"2024-03-17T23:55:01.888676+0000","last_active":"2024-03-17T23:55:04.660140+0000","last_peered":"2024-03-17T23:55:04.660140+0000","last_clean":"2024-03-17T23:55:04.660140+0000","last_became_active":"2024-03-17T23:55:01.888224+0000","last_became_peered":"2024-03-17T23:55:01.888224+0000","last_unstale":"2024-03-17T23:55:04.660140+0000","last_undegraded":"2024-03-17T23:55:04.660140+0000","last_fullsized":"2024-03-17T23:55:04.660140+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:22.075096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:26.020005+0000","last_change":"2024-03-17T23:54:55.857947+0000","last_active":"2024-03-17T23:55:26.020005+0000","last_peered":"2024-03-17T23:55:26.020005+0000","last_clean":"2024-03-17T23:55:26.020005+0000","last_became_active":"2024-03-17T23:54:55.857595+0000","last_became_peered":"2024-03-17T23:54:55.857595+0000","last_unstale":"2024-03-17T23:55:26.020005+0000","last_undegraded":"2024-03-17T23:55:26.020005+0000","last_fullsized":"2024-03-17T23:55:26.020005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:25.433053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306866+0000","last_change":"2024-03-17T23:54:56.347220+0000","last_active":"2024-03-17T23:55:03.306866+0000","last_peered":"2024-03-17T23:55:03.306866+0000","last_clean":"2024-03-17T23:55:03.306866+0000","last_became_active":"2024-03-17T23:54:56.346600+0000","last_became_peered":"2024-03-17T23:54:56.346600+0000","last_unstale":"2024-03-17T23:55:03.306866+0000","last_undegraded":"2024-03-17T23:55:03.306866+0000","last_fullsized":"2024-03-17T23:55:03.306866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:38:25.575448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.452597+0000","last_change":"2024-03-17T23:55:01.887016+0000","last_active":"2024-03-17T23:55:04.452597+0000","last_peered":"2024-03-17T23:55:04.452597+0000","last_clean":"2024-03-17T23:55:04.452597+0000","last_became_active":"2024-03-17T23:55:01.886576+0000","last_became_peered":"2024-03-17T23:55:01.886576+0000","last_unstale":"2024-03-17T23:55:04.452597+0000","last_undegraded":"2024-03-17T23:55:04.452597+0000","last_fullsized":"2024-03-17T23:55:04.452597+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:41:26.995982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.045240+0000","last_change":"2024-03-17T23:54:55.840254+0000","last_active":"2024-03-17T23:55:28.045240+0000","last_peered":"2024-03-17T23:55:28.045240+0000","last_clean":"2024-03-17T23:55:28.045240+0000","last_became_active":"2024-03-17T23:54:55.839740+0000","last_became_peered":"2024-03-17T23:54:55.839740+0000","last_unstale":"2024-03-17T23:55:28.045240+0000","last_undegraded":"2024-03-17T23:55:28.045240+0000","last_fullsized":"2024-03-17T23:55:28.045240+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:39.664083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306815+0000","last_change":"2024-03-17T23:54:56.343262+0000","last_active":"2024-03-17T23:55:03.306815+0000","last_peered":"2024-03-17T23:55:03.306815+0000","last_clean":"2024-03-17T23:55:03.306815+0000","last_became_active":"2024-03-17T23:54:56.342924+0000","last_became_peered":"2024-03-17T23:54:56.342924+0000","last_unstale":"2024-03-17T23:55:03.306815+0000","last_undegraded":"2024-03-17T23:55:03.306815+0000","last_fullsized":"2024-03-17T23:55:03.306815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:35:19.074001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.860057+0000","last_change":"2024-03-17T23:55:01.885081+0000","last_active":"2024-03-17T23:55:02.860057+0000","last_peered":"2024-03-17T23:55:02.860057+0000","last_clean":"2024-03-17T23:55:02.860057+0000","last_became_active":"2024-03-17T23:55:01.883328+0000","last_became_peered":"2024-03-17T23:55:01.883328+0000","last_unstale":"2024-03-17T23:55:02.860057+0000","last_undegraded":"2024-03-17T23:55:02.860057+0000","last_fullsized":"2024-03-17T23:55:02.860057+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:12:10.206774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.032590+0000","last_change":"2024-03-17T23:54:55.852560+0000","last_active":"2024-03-17T23:55:27.032590+0000","last_peered":"2024-03-17T23:55:27.032590+0000","last_clean":"2024-03-17T23:55:27.032590+0000","last_became_active":"2024-03-17T23:54:55.852234+0000","last_became_peered":"2024-03-17T23:54:55.852234+0000","last_unstale":"2024-03-17T23:55:27.032590+0000","last_undegraded":"2024-03-17T23:55:27.032590+0000","last_fullsized":"2024-03-17T23:55:27.032590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:36:22.272022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853988+0000","last_change":"2024-03-17T23:54:56.345946+0000","last_active":"2024-03-17T23:55:02.853988+0000","last_peered":"2024-03-17T23:55:02.853988+0000","last_clean":"2024-03-17T23:55:02.853988+0000","last_became_active":"2024-03-17T23:54:56.345620+0000","last_became_peered":"2024-03-17T23:54:56.345620+0000","last_unstale":"2024-03-17T23:55:02.853988+0000","last_undegraded":"2024-03-17T23:55:02.853988+0000","last_fullsized":"2024-03-17T23:55:02.853988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:03:51.327966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854049+0000","last_change":"2024-03-17T23:55:01.884819+0000","last_active":"2024-03-17T23:55:02.854049+0000","last_peered":"2024-03-17T23:55:02.854049+0000","last_clean":"2024-03-17T23:55:02.854049+0000","last_became_active":"2024-03-17T23:55:01.884045+0000","last_became_peered":"2024-03-17T23:55:01.884045+0000","last_unstale":"2024-03-17T23:55:02.854049+0000","last_undegraded":"2024-03-17T23:55:02.854049+0000","last_fullsized":"2024-03-17T23:55:02.854049+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:10.147132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:26.991053+0000","last_change":"2024-03-17T23:54:55.840370+0000","last_active":"2024-03-17T23:55:26.991053+0000","last_peered":"2024-03-17T23:55:26.991053+0000","last_clean":"2024-03-17T23:55:26.991053+0000","last_became_active":"2024-03-17T23:54:55.840049+0000","last_became_peered":"2024-03-17T23:54:55.840049+0000","last_unstale":"2024-03-17T23:55:26.991053+0000","last_undegraded":"2024-03-17T23:55:26.991053+0000","last_fullsized":"2024-03-17T23:55:26.991053+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:40:46.523632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659909+0000","last_change":"2024-03-17T23:54:56.346533+0000","last_active":"2024-03-17T23:55:04.659909+0000","last_peered":"2024-03-17T23:55:04.659909+0000","last_clean":"2024-03-17T23:55:04.659909+0000","last_became_active":"2024-03-17T23:54:56.346214+0000","last_became_peered":"2024-03-17T23:54:56.346214+0000","last_unstale":"2024-03-17T23:55:04.659909+0000","last_undegraded":"2024-03-17T23:55:04.659909+0000","last_fullsized":"2024-03-17T23:55:04.659909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:06:59.017416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.178391+0000","last_change":"2024-03-17T23:55:02.318944+0000","last_active":"2024-03-17T23:55:03.178391+0000","last_peered":"2024-03-17T23:55:03.178391+0000","last_clean":"2024-03-17T23:55:03.178391+0000","last_became_active":"2024-03-17T23:55:02.318432+0000","last_became_peered":"2024-03-17T23:55:02.318432+0000","last_unstale":"2024-03-17T23:55:03.178391+0000","last_undegraded":"2024-03-17T23:55:03.178391+0000","last_fullsized":"2024-03-17T23:55:03.178391+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:23:47.986701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.452001+0000","last_change":"2024-03-17T23:54:55.873746+0000","last_active":"2024-03-17T23:55:04.452001+0000","last_peered":"2024-03-17T23:55:04.452001+0000","last_clean":"2024-03-17T23:55:04.452001+0000","last_became_active":"2024-03-17T23:54:55.872229+0000","last_became_peered":"2024-03-17T23:54:55.872229+0000","last_unstale":"2024-03-17T23:55:04.452001+0000","last_undegraded":"2024-03-17T23:55:04.452001+0000","last_fullsized":"2024-03-17T23:55:04.452001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:10.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659774+0000","last_change":"2024-03-17T23:54:56.361294+0000","last_active":"2024-03-17T23:55:04.659774+0000","last_peered":"2024-03-17T23:55:04.659774+0000","last_clean":"2024-03-17T23:55:04.659774+0000","last_became_active":"2024-03-17T23:54:56.360976+0000","last_became_peered":"2024-03-17T23:54:56.360976+0000","last_unstale":"2024-03-17T23:55:04.659774+0000","last_undegraded":"2024-03-17T23:55:04.659774+0000","last_fullsized":"2024-03-17T23:55:04.659774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:28:58.685991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.858765+0000","last_change":"2024-03-17T23:55:01.880221+0000","last_active":"2024-03-17T23:55:02.858765+0000","last_peered":"2024-03-17T23:55:02.858765+0000","last_clean":"2024-03-17T23:55:02.858765+0000","last_became_active":"2024-03-17T23:55:01.879785+0000","last_became_peered":"2024-03-17T23:55:01.879785+0000","last_unstale":"2024-03-17T23:55:02.858765+0000","last_undegraded":"2024-03-17T23:55:02.858765+0000","last_fullsized":"2024-03-17T23:55:02.858765+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:24:51.698857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.042458+0000","last_change":"2024-03-17T23:54:55.854176+0000","last_active":"2024-03-17T23:55:28.042458+0000","last_peered":"2024-03-17T23:55:28.042458+0000","last_clean":"2024-03-17T23:55:28.042458+0000","last_became_active":"2024-03-17T23:54:55.853859+0000","last_became_peered":"2024-03-17T23:54:55.853859+0000","last_unstale":"2024-03-17T23:55:28.042458+0000","last_undegraded":"2024-03-17T23:55:28.042458+0000","last_fullsized":"2024-03-17T23:55:28.042458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:43:43.319635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659587+0000","last_change":"2024-03-17T23:54:56.352800+0000","last_active":"2024-03-17T23:55:04.659587+0000","last_peered":"2024-03-17T23:55:04.659587+0000","last_clean":"2024-03-17T23:55:04.659587+0000","last_became_active":"2024-03-17T23:54:56.352466+0000","last_became_peered":"2024-03-17T23:54:56.352466+0000","last_unstale":"2024-03-17T23:55:04.659587+0000","last_undegraded":"2024-03-17T23:55:04.659587+0000","last_fullsized":"2024-03-17T23:55:04.659587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:33:34.895301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.859987+0000","last_change":"2024-03-17T23:55:02.320494+0000","last_active":"2024-03-17T23:55:02.859987+0000","last_peered":"2024-03-17T23:55:02.859987+0000","last_clean":"2024-03-17T23:55:02.859987+0000","last_became_active":"2024-03-17T23:55:02.319188+0000","last_became_peered":"2024-03-17T23:55:02.319188+0000","last_unstale":"2024-03-17T23:55:02.859987+0000","last_undegraded":"2024-03-17T23:55:02.859987+0000","last_fullsized":"2024-03-17T23:55:02.859987+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:30:06.348718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854467+0000","last_change":"2024-03-17T23:54:56.340798+0000","last_active":"2024-03-17T23:55:02.854467+0000","last_peered":"2024-03-17T23:55:02.854467+0000","last_clean":"2024-03-17T23:55:02.854467+0000","last_became_active":"2024-03-17T23:54:56.340386+0000","last_became_peered":"2024-03-17T23:54:56.340386+0000","last_unstale":"2024-03-17T23:55:02.854467+0000","last_undegraded":"2024-03-17T23:55:02.854467+0000","last_fullsized":"2024-03-17T23:55:02.854467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:57:58.546464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:25.003386+0000","last_change":"2024-03-17T23:54:55.870475+0000","last_active":"2024-03-17T23:55:25.003386+0000","last_peered":"2024-03-17T23:55:25.003386+0000","last_clean":"2024-03-17T23:55:25.003386+0000","last_became_active":"2024-03-17T23:54:55.870119+0000","last_became_peered":"2024-03-17T23:54:55.870119+0000","last_unstale":"2024-03-17T23:55:25.003386+0000","last_undegraded":"2024-03-17T23:55:25.003386+0000","last_fullsized":"2024-03-17T23:55:25.003386+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:12:15.421117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.032609+0000","last_change":"2024-03-17T23:55:01.892759+0000","last_active":"2024-03-17T23:55:04.032609+0000","last_peered":"2024-03-17T23:55:04.032609+0000","last_clean":"2024-03-17T23:55:04.032609+0000","last_became_active":"2024-03-17T23:55:01.892240+0000","last_became_peered":"2024-03-17T23:55:01.892240+0000","last_unstale":"2024-03-17T23:55:04.032609+0000","last_undegraded":"2024-03-17T23:55:04.032609+0000","last_fullsized":"2024-03-17T23:55:04.032609+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:16:24.391971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.371961+0000","last_change":"2024-03-17T23:54:56.354898+0000","last_active":"2024-03-17T23:55:03.371961+0000","last_peered":"2024-03-17T23:55:03.371961+0000","last_clean":"2024-03-17T23:55:03.371961+0000","last_became_active":"2024-03-17T23:54:56.354345+0000","last_became_peered":"2024-03-17T23:54:56.354345+0000","last_unstale":"2024-03-17T23:55:03.371961+0000","last_undegraded":"2024-03-17T23:55:03.371961+0000","last_fullsized":"2024-03-17T23:55:03.371961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:18:56.962412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.038111+0000","last_change":"2024-03-17T23:54:55.846706+0000","last_active":"2024-03-17T23:55:28.038111+0000","last_peered":"2024-03-17T23:55:28.038111+0000","last_clean":"2024-03-17T23:55:28.038111+0000","last_became_active":"2024-03-17T23:54:55.846372+0000","last_became_peered":"2024-03-17T23:54:55.846372+0000","last_unstale":"2024-03-17T23:55:28.038111+0000","last_undegraded":"2024-03-17T23:55:28.038111+0000","last_fullsized":"2024-03-17T23:55:28.038111+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:28:47.697975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306579+0000","last_change":"2024-03-17T23:53:34.383413+0000","last_active":"2024-03-17T23:55:03.306579+0000","last_peered":"2024-03-17T23:55:03.306579+0000","last_clean":"2024-03-17T23:55:03.306579+0000","last_became_active":"2024-03-17T23:53:34.382875+0000","last_became_peered":"2024-03-17T23:53:34.382875+0000","last_unstale":"2024-03-17T23:55:03.306579+0000","last_undegraded":"2024-03-17T23:55:03.306579+0000","last_fullsized":"2024-03-17T23:55:03.306579+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:38.641018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:38.641018+0000","last_clean_scrub_stamp":"2024-03-17T23:49:38.641018+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:49:45.234628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854345+0000","last_change":"2024-03-17T23:55:02.319161+0000","last_active":"2024-03-17T23:55:02.854345+0000","last_peered":"2024-03-17T23:55:02.854345+0000","last_clean":"2024-03-17T23:55:02.854345+0000","last_became_active":"2024-03-17T23:55:02.318561+0000","last_became_peered":"2024-03-17T23:55:02.318561+0000","last_unstale":"2024-03-17T23:55:02.854345+0000","last_undegraded":"2024-03-17T23:55:02.854345+0000","last_fullsized":"2024-03-17T23:55:02.854345+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:10.585085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.857455+0000","last_change":"2024-03-17T23:54:56.346097+0000","last_active":"2024-03-17T23:55:02.857455+0000","last_peered":"2024-03-17T23:55:02.857455+0000","last_clean":"2024-03-17T23:55:02.857455+0000","last_became_active":"2024-03-17T23:54:56.345762+0000","last_became_peered":"2024-03-17T23:54:56.345762+0000","last_unstale":"2024-03-17T23:55:02.857455+0000","last_undegraded":"2024-03-17T23:55:02.857455+0000","last_fullsized":"2024-03-17T23:55:02.857455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:43:15.860897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:24.996120+0000","last_change":"2024-03-17T23:54:55.840743+0000","last_active":"2024-03-17T23:55:24.996120+0000","last_peered":"2024-03-17T23:55:24.996120+0000","last_clean":"2024-03-17T23:55:24.996120+0000","last_became_active":"2024-03-17T23:54:55.840463+0000","last_became_peered":"2024-03-17T23:54:55.840463+0000","last_unstale":"2024-03-17T23:55:24.996120+0000","last_undegraded":"2024-03-17T23:55:24.996120+0000","last_fullsized":"2024-03-17T23:55:24.996120+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:24:24.130911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660318+0000","last_change":"2024-03-17T23:55:01.876895+0000","last_active":"2024-03-17T23:55:04.660318+0000","last_peered":"2024-03-17T23:55:04.660318+0000","last_clean":"2024-03-17T23:55:04.660318+0000","last_became_active":"2024-03-17T23:55:01.876443+0000","last_became_peered":"2024-03-17T23:55:01.876443+0000","last_unstale":"2024-03-17T23:55:04.660318+0000","last_undegraded":"2024-03-17T23:55:04.660318+0000","last_fullsized":"2024-03-17T23:55:04.660318+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:51:55.737167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.034886+0000","last_change":"2024-03-17T23:54:55.863601+0000","last_active":"2024-03-17T23:55:27.034886+0000","last_peered":"2024-03-17T23:55:27.034886+0000","last_clean":"2024-03-17T23:55:27.034886+0000","last_became_active":"2024-03-17T23:54:55.863247+0000","last_became_peered":"2024-03-17T23:54:55.863247+0000","last_unstale":"2024-03-17T23:55:27.034886+0000","last_undegraded":"2024-03-17T23:55:27.034886+0000","last_fullsized":"2024-03-17T23:55:27.034886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:53:27.747199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.859188+0000","last_change":"2024-03-17T23:54:56.365913+0000","last_active":"2024-03-17T23:55:02.859188+0000","last_peered":"2024-03-17T23:55:02.859188+0000","last_clean":"2024-03-17T23:55:02.859188+0000","last_became_active":"2024-03-17T23:54:56.365578+0000","last_became_peered":"2024-03-17T23:54:56.365578+0000","last_unstale":"2024-03-17T23:55:02.859188+0000","last_undegraded":"2024-03-17T23:55:02.859188+0000","last_fullsized":"2024-03-17T23:55:02.859188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:49.330026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.371675+0000","last_change":"2024-03-17T23:55:02.319060+0000","last_active":"2024-03-17T23:55:03.371675+0000","last_peered":"2024-03-17T23:55:03.371675+0000","last_clean":"2024-03-17T23:55:03.371675+0000","last_became_active":"2024-03-17T23:55:02.318542+0000","last_became_peered":"2024-03-17T23:55:02.318542+0000","last_unstale":"2024-03-17T23:55:03.371675+0000","last_undegraded":"2024-03-17T23:55:03.371675+0000","last_fullsized":"2024-03-17T23:55:03.371675+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:59:27.334039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'9","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.318574+0000","last_change":"2024-03-17T23:54:55.861011+0000","last_active":"2024-03-17T23:55:27.318574+0000","last_peered":"2024-03-17T23:55:27.318574+0000","last_clean":"2024-03-17T23:55:27.318574+0000","last_became_active":"2024-03-17T23:54:55.860388+0000","last_became_peered":"2024-03-17T23:54:55.860388+0000","last_unstale":"2024-03-17T23:55:27.318574+0000","last_undegraded":"2024-03-17T23:55:27.318574+0000","last_fullsized":"2024-03-17T23:55:27.318574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:20:11.943034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.372867+0000","last_change":"2024-03-17T23:54:56.357572+0000","last_active":"2024-03-17T23:55:03.372867+0000","last_peered":"2024-03-17T23:55:03.372867+0000","last_clean":"2024-03-17T23:55:03.372867+0000","last_became_active":"2024-03-17T23:54:56.357226+0000","last_became_peered":"2024-03-17T23:54:56.357226+0000","last_unstale":"2024-03-17T23:55:03.372867+0000","last_undegraded":"2024-03-17T23:55:03.372867+0000","last_fullsized":"2024-03-17T23:55:03.372867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:35:44.939454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207294+0000","last_change":"2024-03-17T23:55:01.860239+0000","last_active":"2024-03-17T23:55:04.207294+0000","last_peered":"2024-03-17T23:55:04.207294+0000","last_clean":"2024-03-17T23:55:04.207294+0000","last_became_active":"2024-03-17T23:55:01.859754+0000","last_became_peered":"2024-03-17T23:55:01.859754+0000","last_unstale":"2024-03-17T23:55:04.207294+0000","last_undegraded":"2024-03-17T23:55:04.207294+0000","last_fullsized":"2024-03-17T23:55:04.207294+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:30:16.833040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.034726+0000","last_change":"2024-03-17T23:54:55.855917+0000","last_active":"2024-03-17T23:55:27.034726+0000","last_peered":"2024-03-17T23:55:27.034726+0000","last_clean":"2024-03-17T23:55:27.034726+0000","last_became_active":"2024-03-17T23:54:55.855562+0000","last_became_peered":"2024-03-17T23:54:55.855562+0000","last_unstale":"2024-03-17T23:55:27.034726+0000","last_undegraded":"2024-03-17T23:55:27.034726+0000","last_fullsized":"2024-03-17T23:55:27.034726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:44.928086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660707+0000","last_change":"2024-03-17T23:54:56.362634+0000","last_active":"2024-03-17T23:55:04.660707+0000","last_peered":"2024-03-17T23:55:04.660707+0000","last_clean":"2024-03-17T23:55:04.660707+0000","last_became_active":"2024-03-17T23:54:56.361910+0000","last_became_peered":"2024-03-17T23:54:56.361910+0000","last_unstale":"2024-03-17T23:55:04.660707+0000","last_undegraded":"2024-03-17T23:55:04.660707+0000","last_fullsized":"2024-03-17T23:55:04.660707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:20:34.988074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222412+0000","last_change":"2024-03-17T23:55:02.319090+0000","last_active":"2024-03-17T23:55:03.222412+0000","last_peered":"2024-03-17T23:55:03.222412+0000","last_clean":"2024-03-17T23:55:03.222412+0000","last_became_active":"2024-03-17T23:55:02.318695+0000","last_became_peered":"2024-03-17T23:55:02.318695+0000","last_unstale":"2024-03-17T23:55:03.222412+0000","last_undegraded":"2024-03-17T23:55:03.222412+0000","last_fullsized":"2024-03-17T23:55:03.222412+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:32:13.984797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:23.988273+0000","last_change":"2024-03-17T23:54:55.852999+0000","last_active":"2024-03-17T23:55:23.988273+0000","last_peered":"2024-03-17T23:55:23.988273+0000","last_clean":"2024-03-17T23:55:23.988273+0000","last_became_active":"2024-03-17T23:54:55.852653+0000","last_became_peered":"2024-03-17T23:54:55.852653+0000","last_unstale":"2024-03-17T23:55:23.988273+0000","last_undegraded":"2024-03-17T23:55:23.988273+0000","last_fullsized":"2024-03-17T23:55:23.988273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:04:10.561807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.177862+0000","last_change":"2024-03-17T23:54:56.345171+0000","last_active":"2024-03-17T23:55:03.177862+0000","last_peered":"2024-03-17T23:55:03.177862+0000","last_clean":"2024-03-17T23:55:03.177862+0000","last_became_active":"2024-03-17T23:54:56.344146+0000","last_became_peered":"2024-03-17T23:54:56.344146+0000","last_unstale":"2024-03-17T23:55:03.177862+0000","last_undegraded":"2024-03-17T23:55:03.177862+0000","last_fullsized":"2024-03-17T23:55:03.177862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:08.546622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.452941+0000","last_change":"2024-03-17T23:55:01.882655+0000","last_active":"2024-03-17T23:55:04.452941+0000","last_peered":"2024-03-17T23:55:04.452941+0000","last_clean":"2024-03-17T23:55:04.452941+0000","last_became_active":"2024-03-17T23:55:01.882147+0000","last_became_peered":"2024-03-17T23:55:01.882147+0000","last_unstale":"2024-03-17T23:55:04.452941+0000","last_undegraded":"2024-03-17T23:55:04.452941+0000","last_fullsized":"2024-03-17T23:55:04.452941+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:17:55.653103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:26.023625+0000","last_change":"2024-03-17T23:54:55.862499+0000","last_active":"2024-03-17T23:55:26.023625+0000","last_peered":"2024-03-17T23:55:26.023625+0000","last_clean":"2024-03-17T23:55:26.023625+0000","last_became_active":"2024-03-17T23:54:55.862003+0000","last_became_peered":"2024-03-17T23:54:55.862003+0000","last_unstale":"2024-03-17T23:55:26.023625+0000","last_undegraded":"2024-03-17T23:55:26.023625+0000","last_fullsized":"2024-03-17T23:55:26.023625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:37:19.046543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659875+0000","last_change":"2024-03-17T23:54:56.352154+0000","last_active":"2024-03-17T23:55:04.659875+0000","last_peered":"2024-03-17T23:55:04.659875+0000","last_clean":"2024-03-17T23:55:04.659875+0000","last_became_active":"2024-03-17T23:54:56.351818+0000","last_became_peered":"2024-03-17T23:54:56.351818+0000","last_unstale":"2024-03-17T23:55:04.659875+0000","last_undegraded":"2024-03-17T23:55:04.659875+0000","last_fullsized":"2024-03-17T23:55:04.659875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:44:46.538778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.338498+0000","last_change":"2024-03-17T23:55:01.876605+0000","last_active":"2024-03-17T23:55:04.338498+0000","last_peered":"2024-03-17T23:55:04.338498+0000","last_clean":"2024-03-17T23:55:04.338498+0000","last_became_active":"2024-03-17T23:55:01.876253+0000","last_became_peered":"2024-03-17T23:55:01.876253+0000","last_unstale":"2024-03-17T23:55:04.338498+0000","last_undegraded":"2024-03-17T23:55:04.338498+0000","last_fullsized":"2024-03-17T23:55:04.338498+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:27:43.271798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:26.019930+0000","last_change":"2024-03-17T23:54:55.862209+0000","last_active":"2024-03-17T23:55:26.019930+0000","last_peered":"2024-03-17T23:55:26.019930+0000","last_clean":"2024-03-17T23:55:26.019930+0000","last_became_active":"2024-03-17T23:54:55.861829+0000","last_became_peered":"2024-03-17T23:54:55.861829+0000","last_unstale":"2024-03-17T23:55:26.019930+0000","last_undegraded":"2024-03-17T23:55:26.019930+0000","last_fullsized":"2024-03-17T23:55:26.019930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:13:58.756538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.372408+0000","last_change":"2024-03-17T23:54:56.366869+0000","last_active":"2024-03-17T23:55:03.372408+0000","last_peered":"2024-03-17T23:55:03.372408+0000","last_clean":"2024-03-17T23:55:03.372408+0000","last_became_active":"2024-03-17T23:54:56.365380+0000","last_became_peered":"2024-03-17T23:54:56.365380+0000","last_unstale":"2024-03-17T23:55:03.372408+0000","last_undegraded":"2024-03-17T23:55:03.372408+0000","last_fullsized":"2024-03-17T23:55:03.372408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:28:03.010849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222441+0000","last_change":"2024-03-17T23:55:02.318820+0000","last_active":"2024-03-17T23:55:03.222441+0000","last_peered":"2024-03-17T23:55:03.222441+0000","last_clean":"2024-03-17T23:55:03.222441+0000","last_became_active":"2024-03-17T23:55:02.318308+0000","last_became_peered":"2024-03-17T23:55:02.318308+0000","last_unstale":"2024-03-17T23:55:03.222441+0000","last_undegraded":"2024-03-17T23:55:03.222441+0000","last_fullsized":"2024-03-17T23:55:03.222441+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:40:35.224105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:23.987107+0000","last_change":"2024-03-17T23:54:55.837980+0000","last_active":"2024-03-17T23:55:23.987107+0000","last_peered":"2024-03-17T23:55:23.987107+0000","last_clean":"2024-03-17T23:55:23.987107+0000","last_became_active":"2024-03-17T23:54:55.837614+0000","last_became_peered":"2024-03-17T23:54:55.837614+0000","last_unstale":"2024-03-17T23:55:23.987107+0000","last_undegraded":"2024-03-17T23:55:23.987107+0000","last_fullsized":"2024-03-17T23:55:23.987107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:05:07.228281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306819+0000","last_change":"2024-03-17T23:54:56.348534+0000","last_active":"2024-03-17T23:55:03.306819+0000","last_peered":"2024-03-17T23:55:03.306819+0000","last_clean":"2024-03-17T23:55:03.306819+0000","last_became_active":"2024-03-17T23:54:56.348138+0000","last_became_peered":"2024-03-17T23:54:56.348138+0000","last_unstale":"2024-03-17T23:55:03.306819+0000","last_undegraded":"2024-03-17T23:55:03.306819+0000","last_fullsized":"2024-03-17T23:55:03.306819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:47:49.481608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853635+0000","last_change":"2024-03-17T23:55:01.876287+0000","last_active":"2024-03-17T23:55:02.853635+0000","last_peered":"2024-03-17T23:55:02.853635+0000","last_clean":"2024-03-17T23:55:02.853635+0000","last_became_active":"2024-03-17T23:55:01.875861+0000","last_became_peered":"2024-03-17T23:55:01.875861+0000","last_unstale":"2024-03-17T23:55:02.853635+0000","last_undegraded":"2024-03-17T23:55:02.853635+0000","last_fullsized":"2024-03-17T23:55:02.853635+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:33:59.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.042437+0000","last_change":"2024-03-17T23:54:55.870989+0000","last_active":"2024-03-17T23:55:28.042437+0000","last_peered":"2024-03-17T23:55:28.042437+0000","last_clean":"2024-03-17T23:55:28.042437+0000","last_became_active":"2024-03-17T23:54:55.870633+0000","last_became_peered":"2024-03-17T23:54:55.870633+0000","last_unstale":"2024-03-17T23:55:28.042437+0000","last_undegraded":"2024-03-17T23:55:28.042437+0000","last_fullsized":"2024-03-17T23:55:28.042437+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:55.310289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.307089+0000","last_change":"2024-03-17T23:54:56.345874+0000","last_active":"2024-03-17T23:55:03.307089+0000","last_peered":"2024-03-17T23:55:03.307089+0000","last_clean":"2024-03-17T23:55:03.307089+0000","last_became_active":"2024-03-17T23:54:56.345547+0000","last_became_peered":"2024-03-17T23:54:56.345547+0000","last_unstale":"2024-03-17T23:55:03.307089+0000","last_undegraded":"2024-03-17T23:55:03.307089+0000","last_fullsized":"2024-03-17T23:55:03.307089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:52:31.724224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.663521+0000","last_change":"2024-03-17T23:55:01.892947+0000","last_active":"2024-03-17T23:55:04.663521+0000","last_peered":"2024-03-17T23:55:04.663521+0000","last_clean":"2024-03-17T23:55:04.663521+0000","last_became_active":"2024-03-17T23:55:01.892481+0000","last_became_peered":"2024-03-17T23:55:01.892481+0000","last_unstale":"2024-03-17T23:55:04.663521+0000","last_undegraded":"2024-03-17T23:55:04.663521+0000","last_fullsized":"2024-03-17T23:55:04.663521+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:07:14.901490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:26.019965+0000","last_change":"2024-03-17T23:54:55.857943+0000","last_active":"2024-03-17T23:55:26.019965+0000","last_peered":"2024-03-17T23:55:26.019965+0000","last_clean":"2024-03-17T23:55:26.019965+0000","last_became_active":"2024-03-17T23:54:55.857592+0000","last_became_peered":"2024-03-17T23:54:55.857592+0000","last_unstale":"2024-03-17T23:55:26.019965+0000","last_undegraded":"2024-03-17T23:55:26.019965+0000","last_fullsized":"2024-03-17T23:55:26.019965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:50:03.643518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306854+0000","last_change":"2024-03-17T23:54:56.357496+0000","last_active":"2024-03-17T23:55:03.306854+0000","last_peered":"2024-03-17T23:55:03.306854+0000","last_clean":"2024-03-17T23:55:03.306854+0000","last_became_active":"2024-03-17T23:54:56.357242+0000","last_became_peered":"2024-03-17T23:54:56.357242+0000","last_unstale":"2024-03-17T23:55:03.306854+0000","last_undegraded":"2024-03-17T23:55:03.306854+0000","last_fullsized":"2024-03-17T23:55:03.306854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:52:49.941623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853798+0000","last_change":"2024-03-17T23:55:01.866288+0000","last_active":"2024-03-17T23:55:02.853798+0000","last_peered":"2024-03-17T23:55:02.853798+0000","last_clean":"2024-03-17T23:55:02.853798+0000","last_became_active":"2024-03-17T23:55:01.865875+0000","last_became_peered":"2024-03-17T23:55:01.865875+0000","last_unstale":"2024-03-17T23:55:02.853798+0000","last_undegraded":"2024-03-17T23:55:02.853798+0000","last_fullsized":"2024-03-17T23:55:02.853798+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:45:43.198830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.036707+0000","last_change":"2024-03-17T23:54:55.871971+0000","last_active":"2024-03-17T23:55:27.036707+0000","last_peered":"2024-03-17T23:55:27.036707+0000","last_clean":"2024-03-17T23:55:27.036707+0000","last_became_active":"2024-03-17T23:54:55.871434+0000","last_became_peered":"2024-03-17T23:54:55.871434+0000","last_unstale":"2024-03-17T23:55:27.036707+0000","last_undegraded":"2024-03-17T23:55:27.036707+0000","last_fullsized":"2024-03-17T23:55:27.036707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:41:10.667628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.307065+0000","last_change":"2024-03-17T23:54:56.343595+0000","last_active":"2024-03-17T23:55:03.307065+0000","last_peered":"2024-03-17T23:55:03.307065+0000","last_clean":"2024-03-17T23:55:03.307065+0000","last_became_active":"2024-03-17T23:54:56.343270+0000","last_became_peered":"2024-03-17T23:54:56.343270+0000","last_unstale":"2024-03-17T23:55:03.307065+0000","last_undegraded":"2024-03-17T23:55:03.307065+0000","last_fullsized":"2024-03-17T23:55:03.307065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:22:05.904023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659693+0000","last_change":"2024-03-17T23:54:56.346571+0000","last_active":"2024-03-17T23:55:04.659693+0000","last_peered":"2024-03-17T23:55:04.659693+0000","last_clean":"2024-03-17T23:55:04.659693+0000","last_became_active":"2024-03-17T23:54:56.346234+0000","last_became_peered":"2024-03-17T23:54:56.346234+0000","last_unstale":"2024-03-17T23:55:04.659693+0000","last_undegraded":"2024-03-17T23:55:04.659693+0000","last_fullsized":"2024-03-17T23:55:04.659693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:25.448506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207633+0000","last_change":"2024-03-17T23:54:56.342299+0000","last_active":"2024-03-17T23:55:04.207633+0000","last_peered":"2024-03-17T23:55:04.207633+0000","last_clean":"2024-03-17T23:55:04.207633+0000","last_became_active":"2024-03-17T23:54:56.341988+0000","last_became_peered":"2024-03-17T23:54:56.341988+0000","last_unstale":"2024-03-17T23:55:04.207633+0000","last_undegraded":"2024-03-17T23:55:04.207633+0000","last_fullsized":"2024-03-17T23:55:04.207633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:03:38.595595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207300+0000","last_change":"2024-03-17T23:54:56.360601+0000","last_active":"2024-03-17T23:55:04.207300+0000","last_peered":"2024-03-17T23:55:04.207300+0000","last_clean":"2024-03-17T23:55:04.207300+0000","last_became_active":"2024-03-17T23:54:56.360215+0000","last_became_peered":"2024-03-17T23:54:56.360215+0000","last_unstale":"2024-03-17T23:55:04.207300+0000","last_undegraded":"2024-03-17T23:55:04.207300+0000","last_fullsized":"2024-03-17T23:55:04.207300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:42:07.661460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660646+0000","last_change":"2024-03-17T23:54:56.358523+0000","last_active":"2024-03-17T23:55:04.660646+0000","last_peered":"2024-03-17T23:55:04.660646+0000","last_clean":"2024-03-17T23:55:04.660646+0000","last_became_active":"2024-03-17T23:54:56.358172+0000","last_became_peered":"2024-03-17T23:54:56.358172+0000","last_unstale":"2024-03-17T23:55:04.660646+0000","last_undegraded":"2024-03-17T23:55:04.660646+0000","last_fullsized":"2024-03-17T23:55:04.660646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:00.542176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.859048+0000","last_change":"2024-03-17T23:54:56.348619+0000","last_active":"2024-03-17T23:55:02.859048+0000","last_peered":"2024-03-17T23:55:02.859048+0000","last_clean":"2024-03-17T23:55:02.859048+0000","last_became_active":"2024-03-17T23:54:56.348309+0000","last_became_peered":"2024-03-17T23:54:56.348309+0000","last_unstale":"2024-03-17T23:55:02.859048+0000","last_undegraded":"2024-03-17T23:55:02.859048+0000","last_fullsized":"2024-03-17T23:55:02.859048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:38:42.130689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659754+0000","last_change":"2024-03-17T23:54:56.347736+0000","last_active":"2024-03-17T23:55:04.659754+0000","last_peered":"2024-03-17T23:55:04.659754+0000","last_clean":"2024-03-17T23:55:04.659754+0000","last_became_active":"2024-03-17T23:54:56.347283+0000","last_became_peered":"2024-03-17T23:54:56.347283+0000","last_unstale":"2024-03-17T23:55:04.659754+0000","last_undegraded":"2024-03-17T23:55:04.659754+0000","last_fullsized":"2024-03-17T23:55:04.659754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:35:00.683852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207820+0000","last_change":"2024-03-17T23:54:56.344574+0000","last_active":"2024-03-17T23:55:04.207820+0000","last_peered":"2024-03-17T23:55:04.207820+0000","last_clean":"2024-03-17T23:55:04.207820+0000","last_became_active":"2024-03-17T23:54:56.344272+0000","last_became_peered":"2024-03-17T23:54:56.344272+0000","last_unstale":"2024-03-17T23:55:04.207820+0000","last_undegraded":"2024-03-17T23:55:04.207820+0000","last_fullsized":"2024-03-17T23:55:04.207820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:37:37.134152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.372977+0000","last_change":"2024-03-17T23:54:56.365633+0000","last_active":"2024-03-17T23:55:03.372977+0000","last_peered":"2024-03-17T23:55:03.372977+0000","last_clean":"2024-03-17T23:55:03.372977+0000","last_became_active":"2024-03-17T23:54:56.364533+0000","last_became_peered":"2024-03-17T23:54:56.364533+0000","last_unstale":"2024-03-17T23:55:03.372977+0000","last_undegraded":"2024-03-17T23:55:03.372977+0000","last_fullsized":"2024-03-17T23:55:03.372977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:05:20.848672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854152+0000","last_change":"2024-03-17T23:54:56.365614+0000","last_active":"2024-03-17T23:55:02.854152+0000","last_peered":"2024-03-17T23:55:02.854152+0000","last_clean":"2024-03-17T23:55:02.854152+0000","last_became_active":"2024-03-17T23:54:56.365301+0000","last_became_peered":"2024-03-17T23:54:56.365301+0000","last_unstale":"2024-03-17T23:55:02.854152+0000","last_undegraded":"2024-03-17T23:55:02.854152+0000","last_fullsized":"2024-03-17T23:55:02.854152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:20.122964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.371692+0000","last_change":"2024-03-17T23:54:56.365531+0000","last_active":"2024-03-17T23:55:03.371692+0000","last_peered":"2024-03-17T23:55:03.371692+0000","last_clean":"2024-03-17T23:55:03.371692+0000","last_became_active":"2024-03-17T23:54:56.364675+0000","last_became_peered":"2024-03-17T23:54:56.364675+0000","last_unstale":"2024-03-17T23:55:03.371692+0000","last_undegraded":"2024-03-17T23:55:03.371692+0000","last_fullsized":"2024-03-17T23:55:03.371692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:25:49.612055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853681+0000","last_change":"2024-03-17T23:54:56.355306+0000","last_active":"2024-03-17T23:55:02.853681+0000","last_peered":"2024-03-17T23:55:02.853681+0000","last_clean":"2024-03-17T23:55:02.853681+0000","last_became_active":"2024-03-17T23:54:56.354975+0000","last_became_peered":"2024-03-17T23:54:56.354975+0000","last_unstale":"2024-03-17T23:55:02.853681+0000","last_undegraded":"2024-03-17T23:55:02.853681+0000","last_fullsized":"2024-03-17T23:55:02.853681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:58.606691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.858602+0000","last_change":"2024-03-17T23:54:56.350333+0000","last_active":"2024-03-17T23:55:02.858602+0000","last_peered":"2024-03-17T23:55:02.858602+0000","last_clean":"2024-03-17T23:55:02.858602+0000","last_became_active":"2024-03-17T23:54:56.349953+0000","last_became_peered":"2024-03-17T23:54:56.349953+0000","last_unstale":"2024-03-17T23:55:02.858602+0000","last_undegraded":"2024-03-17T23:55:02.858602+0000","last_fullsized":"2024-03-17T23:55:02.858602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:19:34.639852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659730+0000","last_change":"2024-03-17T23:54:56.343143+0000","last_active":"2024-03-17T23:55:04.659730+0000","last_peered":"2024-03-17T23:55:04.659730+0000","last_clean":"2024-03-17T23:55:04.659730+0000","last_became_active":"2024-03-17T23:54:56.342772+0000","last_became_peered":"2024-03-17T23:54:56.342772+0000","last_unstale":"2024-03-17T23:55:04.659730+0000","last_undegraded":"2024-03-17T23:55:04.659730+0000","last_fullsized":"2024-03-17T23:55:04.659730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:33:45.668975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246647,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10565,"num_objects":93,"num_object_clones":0,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":201,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":107913,"data_compressed":3670,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":490682,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645336,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1081182,"data_compressed":50389,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808863,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":494708,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070501,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":490990,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332138,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":1087487,"data_compressed":50389,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":486116,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":529052,"data_compressed":16519,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":487572,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":529663,"data_compressed":16519,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":485802,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":490190,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":1083069,"data_compressed":50389,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34061,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34061,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":43658,"data_compressed":1835,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":44269,"data_compressed":1835,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:55:30.297 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:55:30.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:30 smithi191 ceph-mon[36708]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-17T23:55:30.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:30 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2158995707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:55:30.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:30 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3768899117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:30.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:30 smithi079 ceph-mon[36241]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-17T23:55:30.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:30 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2158995707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:55:30.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:30 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3768899117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:30.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.551+0000 7f9059c00640 1 -- 172.21.15.12:0/3942162078 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90541031d0 msgr2=0x7f9054109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:30.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.551+0000 7f9059c00640 1 --2- 172.21.15.12:0/3942162078 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90541031d0 0x7f9054109a60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f904c009900 tx=0x7f904c02bc30 comp rx=0 tx=0).stop 2024-03-17T23:55:30.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.551+0000 7f9059c00640 1 -- 172.21.15.12:0/3942162078 shutdown_connections 2024-03-17T23:55:30.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.551+0000 7f9059c00640 1 --2- 172.21.15.12:0/3942162078 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90541031d0 0x7f9054109a60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.551+0000 7f9059c00640 1 --2- 172.21.15.12:0/3942162078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9054102810 0x7f9054102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.551+0000 7f9059c00640 1 --2- 172.21.15.12:0/3942162078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9054101620 0x7f9054101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.551+0000 7f9059c00640 1 -- 172.21.15.12:0/3942162078 >> 172.21.15.12:0/3942162078 conn(0x7f90540fcb90 msgr2=0x7f90540fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:30.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.551+0000 7f9059c00640 1 -- 172.21.15.12:0/3942162078 shutdown_connections 2024-03-17T23:55:30.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.551+0000 7f9059c00640 1 -- 172.21.15.12:0/3942162078 wait complete. 2024-03-17T23:55:30.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.552+0000 7f9059c00640 1 Processor -- start 2024-03-17T23:55:30.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.552+0000 7f9059c00640 1 -- start start 2024-03-17T23:55:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.552+0000 7f9059c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9054101620 0x7f9054199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.552+0000 7f9059c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9054102810 0x7f9054199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.552+0000 7f9059c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90541031d0 0x7f90541a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.552+0000 7f9059c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f905410d2d0 con 0x7f9054101620 2024-03-17T23:55:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.553+0000 7f9059c00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f905410d150 con 0x7f90541031d0 2024-03-17T23:55:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.553+0000 7f9059c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f905410d450 con 0x7f9054102810 2024-03-17T23:55:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.553+0000 7f904b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9054102810 0x7f9054199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.553+0000 7f9053e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9054101620 0x7f9054199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.553+0000 7f9059200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90541031d0 0x7f90541a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.553+0000 7f904b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9054102810 0x7f9054199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:34568/0 (socket says 172.21.15.12:34568) 2024-03-17T23:55:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.553+0000 7f904b400640 1 -- 172.21.15.12:0/149916744 learned_addr learned my addr 172.21.15.12:0/149916744 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.553+0000 7f9059200640 1 -- 172.21.15.12:0/149916744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9054102810 msgr2=0x7f9054199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.553+0000 7f9059200640 1 --2- 172.21.15.12:0/149916744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9054102810 0x7f9054199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.553+0000 7f9059200640 1 -- 172.21.15.12:0/149916744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9054101620 msgr2=0x7f9054199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:30.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.553+0000 7f9059200640 1 --2- 172.21.15.12:0/149916744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9054101620 0x7f9054199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.553+0000 7f9059200640 1 -- 172.21.15.12:0/149916744 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90541a1640 con 0x7f90541031d0 2024-03-17T23:55:30.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.553+0000 7f904b400640 1 --2- 172.21.15.12:0/149916744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9054102810 0x7f9054199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:55:30.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.554+0000 7f9059200640 1 --2- 172.21.15.12:0/149916744 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90541031d0 0x7f90541a0fd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f904c02c170 tx=0x7f904c02c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:30.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.554+0000 7f9051600640 1 -- 172.21.15.12:0/149916744 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c043020 con 0x7f90541031d0 2024-03-17T23:55:30.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.554+0000 7f9051600640 1 -- 172.21.15.12:0/149916744 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f904c0054e0 con 0x7f90541031d0 2024-03-17T23:55:30.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.554+0000 7f9051600640 1 -- 172.21.15.12:0/149916744 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c003b40 con 0x7f90541031d0 2024-03-17T23:55:30.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.554+0000 7f9059c00640 1 -- 172.21.15.12:0/149916744 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90541a18d0 con 0x7f90541031d0 2024-03-17T23:55:30.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.554+0000 7f9059c00640 1 -- 172.21.15.12:0/149916744 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90541a1da0 con 0x7f90541031d0 2024-03-17T23:55:30.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.555+0000 7f9059c00640 1 -- 172.21.15.12:0/149916744 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90541a1fa0 con 0x7f90541031d0 2024-03-17T23:55:30.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.557+0000 7f9051600640 1 -- 172.21.15.12:0/149916744 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f904c046020 con 0x7f90541031d0 2024-03-17T23:55:30.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.558+0000 7f9051600640 1 --2- 172.21.15.12:0/149916744 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f90380a2910 0x7f90380a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.558+0000 7f9051600640 1 -- 172.21.15.12:0/149916744 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f904c0f0cb0 con 0x7f90541031d0 2024-03-17T23:55:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.559+0000 7f9053e00640 1 --2- 172.21.15.12:0/149916744 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f90380a2910 0x7f90380a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.559+0000 7f9053e00640 1 --2- 172.21.15.12:0/149916744 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f90380a2910 0x7f90380a4dd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9054102670 tx=0x7f9044009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:30.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.567+0000 7f9051600640 1 -- 172.21.15.12:0/149916744 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f904c04d020 con 0x7f90541031d0 2024-03-17T23:55:30.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.768+0000 7f9059c00640 1 -- 172.21.15.12:0/149916744 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9054075130 con 0x7f90380a2910 2024-03-17T23:55:30.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.783+0000 7f9051600640 1 -- 172.21.15.12:0/149916744 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340718 (secure 0 0 0) 0x7f9054075130 con 0x7f90380a2910 2024-03-17T23:55:30.788 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:30.792 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-17T23:55:30.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.791+0000 7f9059c00640 1 -- 172.21.15.12:0/149916744 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f90380a2910 msgr2=0x7f90380a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:30.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.791+0000 7f9059c00640 1 --2- 172.21.15.12:0/149916744 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f90380a2910 0x7f90380a4dd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9054102670 tx=0x7f9044009500 comp rx=0 tx=0).stop 2024-03-17T23:55:30.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.791+0000 7f9059c00640 1 -- 172.21.15.12:0/149916744 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90541031d0 msgr2=0x7f90541a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:30.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.791+0000 7f9059c00640 1 --2- 172.21.15.12:0/149916744 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90541031d0 0x7f90541a0fd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f904c02c170 tx=0x7f904c02c640 comp rx=0 tx=0).stop 2024-03-17T23:55:30.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.791+0000 7f9059c00640 1 -- 172.21.15.12:0/149916744 shutdown_connections 2024-03-17T23:55:30.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.791+0000 7f9059c00640 1 --2- 172.21.15.12:0/149916744 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f90380a2910 0x7f90380a4dd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.791+0000 7f9059c00640 1 --2- 172.21.15.12:0/149916744 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90541031d0 0x7f90541a0fd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.791+0000 7f9059c00640 1 --2- 172.21.15.12:0/149916744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9054102810 0x7f9054199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.791+0000 7f9059c00640 1 --2- 172.21.15.12:0/149916744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9054101620 0x7f9054199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:30.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.792+0000 7f9059c00640 1 -- 172.21.15.12:0/149916744 >> 172.21.15.12:0/149916744 conn(0x7f90540fcb90 msgr2=0x7f9054073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:30.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.792+0000 7f9059c00640 1 -- 172.21.15.12:0/149916744 shutdown_connections 2024-03-17T23:55:30.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:30.792+0000 7f9059c00640 1 -- 172.21.15.12:0/149916744 wait complete. 2024-03-17T23:55:30.820 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-03-17T23:55:30.108634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601167,"num_objects":97,"num_object_clones":0,"num_object_copies":293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":154,"num_read_kb":133,"num_write":384,"num_write_kb":3009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":316,"ondisk_log_size":316,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":352872,"kb_used_data":12040,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588952,"statfs":{"total":1151940427776,"available":1151579086848,"internally_reserved":0,"allocated":12328960,"data_stored":7736513,"data_compressed":286993,"data_compressed_allocated":389120,"data_compressed_original":2277376,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6300,"num_objects":73,"num_object_clones":0,"num_object_copies":221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":155,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006315"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660413+0000","last_change":"2024-03-17T23:54:56.354062+0000","last_active":"2024-03-17T23:55:04.660413+0000","last_peered":"2024-03-17T23:55:04.660413+0000","last_clean":"2024-03-17T23:55:04.660413+0000","last_became_active":"2024-03-17T23:54:56.353753+0000","last_became_peered":"2024-03-17T23:54:56.353753+0000","last_unstale":"2024-03-17T23:55:04.660413+0000","last_undegraded":"2024-03-17T23:55:04.660413+0000","last_fullsized":"2024-03-17T23:55:04.660413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:41:50.385420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.663922+0000","last_change":"2024-03-17T23:54:56.356507+0000","last_active":"2024-03-17T23:55:04.663922+0000","last_peered":"2024-03-17T23:55:04.663922+0000","last_clean":"2024-03-17T23:55:04.663922+0000","last_became_active":"2024-03-17T23:54:56.355714+0000","last_became_peered":"2024-03-17T23:54:56.355714+0000","last_unstale":"2024-03-17T23:55:04.663922+0000","last_undegraded":"2024-03-17T23:55:04.663922+0000","last_fullsized":"2024-03-17T23:55:04.663922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:44:15.992630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660980+0000","last_change":"2024-03-17T23:54:56.352207+0000","last_active":"2024-03-17T23:55:04.660980+0000","last_peered":"2024-03-17T23:55:04.660980+0000","last_clean":"2024-03-17T23:55:04.660980+0000","last_became_active":"2024-03-17T23:54:56.351704+0000","last_became_peered":"2024-03-17T23:54:56.351704+0000","last_unstale":"2024-03-17T23:55:04.660980+0000","last_undegraded":"2024-03-17T23:55:04.660980+0000","last_fullsized":"2024-03-17T23:55:04.660980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:34.328120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222439+0000","last_change":"2024-03-17T23:54:56.343336+0000","last_active":"2024-03-17T23:55:03.222439+0000","last_peered":"2024-03-17T23:55:03.222439+0000","last_clean":"2024-03-17T23:55:03.222439+0000","last_became_active":"2024-03-17T23:54:56.343033+0000","last_became_peered":"2024-03-17T23:54:56.343033+0000","last_unstale":"2024-03-17T23:55:03.222439+0000","last_undegraded":"2024-03-17T23:55:03.222439+0000","last_fullsized":"2024-03-17T23:55:03.222439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:40:21.814558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.178101+0000","last_change":"2024-03-17T23:54:56.360541+0000","last_active":"2024-03-17T23:55:03.178101+0000","last_peered":"2024-03-17T23:55:03.178101+0000","last_clean":"2024-03-17T23:55:03.178101+0000","last_became_active":"2024-03-17T23:54:56.360204+0000","last_became_peered":"2024-03-17T23:54:56.360204+0000","last_unstale":"2024-03-17T23:55:03.178101+0000","last_undegraded":"2024-03-17T23:55:03.178101+0000","last_fullsized":"2024-03-17T23:55:03.178101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:57:37.853168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659365+0000","last_change":"2024-03-17T23:54:56.363727+0000","last_active":"2024-03-17T23:55:04.659365+0000","last_peered":"2024-03-17T23:55:04.659365+0000","last_clean":"2024-03-17T23:55:04.659365+0000","last_became_active":"2024-03-17T23:54:56.363399+0000","last_became_peered":"2024-03-17T23:54:56.363399+0000","last_unstale":"2024-03-17T23:55:04.659365+0000","last_undegraded":"2024-03-17T23:55:04.659365+0000","last_fullsized":"2024-03-17T23:55:04.659365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:42.451379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306609+0000","last_change":"2024-03-17T23:54:56.361881+0000","last_active":"2024-03-17T23:55:03.306609+0000","last_peered":"2024-03-17T23:55:03.306609+0000","last_clean":"2024-03-17T23:55:03.306609+0000","last_became_active":"2024-03-17T23:54:56.361531+0000","last_became_peered":"2024-03-17T23:54:56.361531+0000","last_unstale":"2024-03-17T23:55:03.306609+0000","last_undegraded":"2024-03-17T23:55:03.306609+0000","last_fullsized":"2024-03-17T23:55:03.306609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:44:39.792763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.371774+0000","last_change":"2024-03-17T23:54:56.358429+0000","last_active":"2024-03-17T23:55:03.371774+0000","last_peered":"2024-03-17T23:55:03.371774+0000","last_clean":"2024-03-17T23:55:03.371774+0000","last_became_active":"2024-03-17T23:54:56.355268+0000","last_became_peered":"2024-03-17T23:54:56.355268+0000","last_unstale":"2024-03-17T23:55:03.371774+0000","last_undegraded":"2024-03-17T23:55:03.371774+0000","last_fullsized":"2024-03-17T23:55:03.371774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:00:37.687542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853925+0000","last_change":"2024-03-17T23:54:56.344228+0000","last_active":"2024-03-17T23:55:02.853925+0000","last_peered":"2024-03-17T23:55:02.853925+0000","last_clean":"2024-03-17T23:55:02.853925+0000","last_became_active":"2024-03-17T23:54:56.343863+0000","last_became_peered":"2024-03-17T23:54:56.343863+0000","last_unstale":"2024-03-17T23:55:02.853925+0000","last_undegraded":"2024-03-17T23:55:02.853925+0000","last_fullsized":"2024-03-17T23:55:02.853925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:45:12.552611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659558+0000","last_change":"2024-03-17T23:54:56.341215+0000","last_active":"2024-03-17T23:55:04.659558+0000","last_peered":"2024-03-17T23:55:04.659558+0000","last_clean":"2024-03-17T23:55:04.659558+0000","last_became_active":"2024-03-17T23:54:56.340850+0000","last_became_peered":"2024-03-17T23:54:56.340850+0000","last_unstale":"2024-03-17T23:55:04.659558+0000","last_undegraded":"2024-03-17T23:55:04.659558+0000","last_fullsized":"2024-03-17T23:55:04.659558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:28.273320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306995+0000","last_change":"2024-03-17T23:54:56.346613+0000","last_active":"2024-03-17T23:55:03.306995+0000","last_peered":"2024-03-17T23:55:03.306995+0000","last_clean":"2024-03-17T23:55:03.306995+0000","last_became_active":"2024-03-17T23:54:56.345907+0000","last_became_peered":"2024-03-17T23:54:56.345907+0000","last_unstale":"2024-03-17T23:55:03.306995+0000","last_undegraded":"2024-03-17T23:55:03.306995+0000","last_fullsized":"2024-03-17T23:55:03.306995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:21.493841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222384+0000","last_change":"2024-03-17T23:54:56.341924+0000","last_active":"2024-03-17T23:55:03.222384+0000","last_peered":"2024-03-17T23:55:03.222384+0000","last_clean":"2024-03-17T23:55:03.222384+0000","last_became_active":"2024-03-17T23:54:56.341469+0000","last_became_peered":"2024-03-17T23:54:56.341469+0000","last_unstale":"2024-03-17T23:55:03.222384+0000","last_undegraded":"2024-03-17T23:55:03.222384+0000","last_fullsized":"2024-03-17T23:55:03.222384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:49:47.256954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.858501+0000","last_change":"2024-03-17T23:54:56.332469+0000","last_active":"2024-03-17T23:55:02.858501+0000","last_peered":"2024-03-17T23:55:02.858501+0000","last_clean":"2024-03-17T23:55:02.858501+0000","last_became_active":"2024-03-17T23:54:56.330684+0000","last_became_peered":"2024-03-17T23:54:56.330684+0000","last_unstale":"2024-03-17T23:55:02.858501+0000","last_undegraded":"2024-03-17T23:55:02.858501+0000","last_fullsized":"2024-03-17T23:55:02.858501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:25.211711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660643+0000","last_change":"2024-03-17T23:54:56.354451+0000","last_active":"2024-03-17T23:55:04.660643+0000","last_peered":"2024-03-17T23:55:04.660643+0000","last_clean":"2024-03-17T23:55:04.660643+0000","last_became_active":"2024-03-17T23:54:56.354160+0000","last_became_peered":"2024-03-17T23:54:56.354160+0000","last_unstale":"2024-03-17T23:55:04.660643+0000","last_undegraded":"2024-03-17T23:55:04.660643+0000","last_fullsized":"2024-03-17T23:55:04.660643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:07:37.644444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660653+0000","last_change":"2024-03-17T23:54:56.353705+0000","last_active":"2024-03-17T23:55:04.660653+0000","last_peered":"2024-03-17T23:55:04.660653+0000","last_clean":"2024-03-17T23:55:04.660653+0000","last_became_active":"2024-03-17T23:54:56.352389+0000","last_became_peered":"2024-03-17T23:54:56.352389+0000","last_unstale":"2024-03-17T23:55:04.660653+0000","last_undegraded":"2024-03-17T23:55:04.660653+0000","last_fullsized":"2024-03-17T23:55:04.660653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:17.153753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853481+0000","last_change":"2024-03-17T23:54:56.348467+0000","last_active":"2024-03-17T23:55:02.853481+0000","last_peered":"2024-03-17T23:55:02.853481+0000","last_clean":"2024-03-17T23:55:02.853481+0000","last_became_active":"2024-03-17T23:54:56.348118+0000","last_became_peered":"2024-03-17T23:54:56.348118+0000","last_unstale":"2024-03-17T23:55:02.853481+0000","last_undegraded":"2024-03-17T23:55:02.853481+0000","last_fullsized":"2024-03-17T23:55:02.853481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:53:45.717303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660631+0000","last_change":"2024-03-17T23:54:56.354117+0000","last_active":"2024-03-17T23:55:04.660631+0000","last_peered":"2024-03-17T23:55:04.660631+0000","last_clean":"2024-03-17T23:55:04.660631+0000","last_became_active":"2024-03-17T23:54:56.353818+0000","last_became_peered":"2024-03-17T23:54:56.353818+0000","last_unstale":"2024-03-17T23:55:04.660631+0000","last_undegraded":"2024-03-17T23:55:04.660631+0000","last_fullsized":"2024-03-17T23:55:04.660631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:09:17.314024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.449909+0000","last_change":"2024-03-17T23:54:56.363781+0000","last_active":"2024-03-17T23:55:04.449909+0000","last_peered":"2024-03-17T23:55:04.449909+0000","last_clean":"2024-03-17T23:55:04.449909+0000","last_became_active":"2024-03-17T23:54:56.363364+0000","last_became_peered":"2024-03-17T23:54:56.363364+0000","last_unstale":"2024-03-17T23:55:04.449909+0000","last_undegraded":"2024-03-17T23:55:04.449909+0000","last_fullsized":"2024-03-17T23:55:04.449909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:21:00.395329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207852+0000","last_change":"2024-03-17T23:54:56.360905+0000","last_active":"2024-03-17T23:55:04.207852+0000","last_peered":"2024-03-17T23:55:04.207852+0000","last_clean":"2024-03-17T23:55:04.207852+0000","last_became_active":"2024-03-17T23:54:56.360492+0000","last_became_peered":"2024-03-17T23:54:56.360492+0000","last_unstale":"2024-03-17T23:55:04.207852+0000","last_undegraded":"2024-03-17T23:55:04.207852+0000","last_fullsized":"2024-03-17T23:55:04.207852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:34:42.440635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207652+0000","last_change":"2024-03-17T23:55:01.889086+0000","last_active":"2024-03-17T23:55:04.207652+0000","last_peered":"2024-03-17T23:55:04.207652+0000","last_clean":"2024-03-17T23:55:04.207652+0000","last_became_active":"2024-03-17T23:55:01.888505+0000","last_became_peered":"2024-03-17T23:55:01.888505+0000","last_unstale":"2024-03-17T23:55:04.207652+0000","last_undegraded":"2024-03-17T23:55:04.207652+0000","last_fullsized":"2024-03-17T23:55:04.207652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:00:43.490360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.042292+0000","last_change":"2024-03-17T23:54:55.858790+0000","last_active":"2024-03-17T23:55:28.042292+0000","last_peered":"2024-03-17T23:55:28.042292+0000","last_clean":"2024-03-17T23:55:28.042292+0000","last_became_active":"2024-03-17T23:54:55.858437+0000","last_became_peered":"2024-03-17T23:54:55.858437+0000","last_unstale":"2024-03-17T23:55:28.042292+0000","last_undegraded":"2024-03-17T23:55:28.042292+0000","last_fullsized":"2024-03-17T23:55:28.042292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:49:26.830471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.372883+0000","last_change":"2024-03-17T23:54:56.356641+0000","last_active":"2024-03-17T23:55:03.372883+0000","last_peered":"2024-03-17T23:55:03.372883+0000","last_clean":"2024-03-17T23:55:03.372883+0000","last_became_active":"2024-03-17T23:54:56.355402+0000","last_became_peered":"2024-03-17T23:54:56.355402+0000","last_unstale":"2024-03-17T23:55:03.372883+0000","last_undegraded":"2024-03-17T23:55:03.372883+0000","last_fullsized":"2024-03-17T23:55:03.372883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:01:04.699357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222348+0000","last_change":"2024-03-17T23:55:01.888632+0000","last_active":"2024-03-17T23:55:03.222348+0000","last_peered":"2024-03-17T23:55:03.222348+0000","last_clean":"2024-03-17T23:55:03.222348+0000","last_became_active":"2024-03-17T23:55:01.888240+0000","last_became_peered":"2024-03-17T23:55:01.888240+0000","last_unstale":"2024-03-17T23:55:03.222348+0000","last_undegraded":"2024-03-17T23:55:03.222348+0000","last_fullsized":"2024-03-17T23:55:03.222348+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:02:04.562112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.372960+0000","last_change":"2024-03-17T23:54:55.871842+0000","last_active":"2024-03-17T23:55:03.372960+0000","last_peered":"2024-03-17T23:55:03.372960+0000","last_clean":"2024-03-17T23:55:03.372960+0000","last_became_active":"2024-03-17T23:54:55.870965+0000","last_became_peered":"2024-03-17T23:54:55.870965+0000","last_unstale":"2024-03-17T23:55:03.372960+0000","last_undegraded":"2024-03-17T23:55:03.372960+0000","last_fullsized":"2024-03-17T23:55:03.372960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:27:04.762902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.661453+0000","last_change":"2024-03-17T23:54:56.368277+0000","last_active":"2024-03-17T23:55:04.661453+0000","last_peered":"2024-03-17T23:55:04.661453+0000","last_clean":"2024-03-17T23:55:04.661453+0000","last_became_active":"2024-03-17T23:54:56.366229+0000","last_became_peered":"2024-03-17T23:54:56.366229+0000","last_unstale":"2024-03-17T23:55:04.661453+0000","last_undegraded":"2024-03-17T23:55:04.661453+0000","last_fullsized":"2024-03-17T23:55:04.661453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:50:05.241745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222480+0000","last_change":"2024-03-17T23:55:01.881731+0000","last_active":"2024-03-17T23:55:03.222480+0000","last_peered":"2024-03-17T23:55:03.222480+0000","last_clean":"2024-03-17T23:55:03.222480+0000","last_became_active":"2024-03-17T23:55:01.881383+0000","last_became_peered":"2024-03-17T23:55:01.881383+0000","last_unstale":"2024-03-17T23:55:03.222480+0000","last_undegraded":"2024-03-17T23:55:03.222480+0000","last_fullsized":"2024-03-17T23:55:03.222480+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:45.095672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.028239+0000","last_change":"2024-03-17T23:54:55.838014+0000","last_active":"2024-03-17T23:55:28.028239+0000","last_peered":"2024-03-17T23:55:28.028239+0000","last_clean":"2024-03-17T23:55:28.028239+0000","last_became_active":"2024-03-17T23:54:55.837694+0000","last_became_peered":"2024-03-17T23:54:55.837694+0000","last_unstale":"2024-03-17T23:55:28.028239+0000","last_undegraded":"2024-03-17T23:55:28.028239+0000","last_fullsized":"2024-03-17T23:55:28.028239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:16:01.173460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853699+0000","last_change":"2024-03-17T23:54:56.343240+0000","last_active":"2024-03-17T23:55:02.853699+0000","last_peered":"2024-03-17T23:55:02.853699+0000","last_clean":"2024-03-17T23:55:02.853699+0000","last_became_active":"2024-03-17T23:54:56.342928+0000","last_became_peered":"2024-03-17T23:54:56.342928+0000","last_unstale":"2024-03-17T23:55:02.853699+0000","last_undegraded":"2024-03-17T23:55:02.853699+0000","last_fullsized":"2024-03-17T23:55:02.853699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:33:43.922886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222266+0000","last_change":"2024-03-17T23:55:01.880207+0000","last_active":"2024-03-17T23:55:03.222266+0000","last_peered":"2024-03-17T23:55:03.222266+0000","last_clean":"2024-03-17T23:55:03.222266+0000","last_became_active":"2024-03-17T23:55:01.879748+0000","last_became_peered":"2024-03-17T23:55:01.879748+0000","last_unstale":"2024-03-17T23:55:03.222266+0000","last_undegraded":"2024-03-17T23:55:03.222266+0000","last_fullsized":"2024-03-17T23:55:03.222266+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:33.782782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.032972+0000","last_change":"2024-03-17T23:54:55.839822+0000","last_active":"2024-03-17T23:55:27.032972+0000","last_peered":"2024-03-17T23:55:27.032972+0000","last_clean":"2024-03-17T23:55:27.032972+0000","last_became_active":"2024-03-17T23:54:55.839506+0000","last_became_peered":"2024-03-17T23:54:55.839506+0000","last_unstale":"2024-03-17T23:55:27.032972+0000","last_undegraded":"2024-03-17T23:55:27.032972+0000","last_fullsized":"2024-03-17T23:55:27.032972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:08.743582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854388+0000","last_change":"2024-03-17T23:54:56.346914+0000","last_active":"2024-03-17T23:55:02.854388+0000","last_peered":"2024-03-17T23:55:02.854388+0000","last_clean":"2024-03-17T23:55:02.854388+0000","last_became_active":"2024-03-17T23:54:56.346620+0000","last_became_peered":"2024-03-17T23:54:56.346620+0000","last_unstale":"2024-03-17T23:55:02.854388+0000","last_undegraded":"2024-03-17T23:55:02.854388+0000","last_fullsized":"2024-03-17T23:55:02.854388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:44:58.643614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207738+0000","last_change":"2024-03-17T23:55:01.864527+0000","last_active":"2024-03-17T23:55:04.207738+0000","last_peered":"2024-03-17T23:55:04.207738+0000","last_clean":"2024-03-17T23:55:04.207738+0000","last_became_active":"2024-03-17T23:55:01.864106+0000","last_became_peered":"2024-03-17T23:55:01.864106+0000","last_unstale":"2024-03-17T23:55:04.207738+0000","last_undegraded":"2024-03-17T23:55:04.207738+0000","last_fullsized":"2024-03-17T23:55:04.207738+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:27:19.811639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:23.988262+0000","last_change":"2024-03-17T23:54:55.842156+0000","last_active":"2024-03-17T23:55:23.988262+0000","last_peered":"2024-03-17T23:55:23.988262+0000","last_clean":"2024-03-17T23:55:23.988262+0000","last_became_active":"2024-03-17T23:54:55.841858+0000","last_became_peered":"2024-03-17T23:54:55.841858+0000","last_unstale":"2024-03-17T23:55:23.988262+0000","last_undegraded":"2024-03-17T23:55:23.988262+0000","last_fullsized":"2024-03-17T23:55:23.988262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:01:05.967638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.178220+0000","last_change":"2024-03-17T23:54:56.357770+0000","last_active":"2024-03-17T23:55:03.178220+0000","last_peered":"2024-03-17T23:55:03.178220+0000","last_clean":"2024-03-17T23:55:03.178220+0000","last_became_active":"2024-03-17T23:54:56.357411+0000","last_became_peered":"2024-03-17T23:54:56.357411+0000","last_unstale":"2024-03-17T23:55:03.178220+0000","last_undegraded":"2024-03-17T23:55:03.178220+0000","last_fullsized":"2024-03-17T23:55:03.178220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:01:04.685138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207636+0000","last_change":"2024-03-17T23:55:01.883104+0000","last_active":"2024-03-17T23:55:04.207636+0000","last_peered":"2024-03-17T23:55:04.207636+0000","last_clean":"2024-03-17T23:55:04.207636+0000","last_became_active":"2024-03-17T23:55:01.882650+0000","last_became_peered":"2024-03-17T23:55:01.882650+0000","last_unstale":"2024-03-17T23:55:04.207636+0000","last_undegraded":"2024-03-17T23:55:04.207636+0000","last_fullsized":"2024-03-17T23:55:04.207636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:54:18.420997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:25.000312+0000","last_change":"2024-03-17T23:54:55.854625+0000","last_active":"2024-03-17T23:55:25.000312+0000","last_peered":"2024-03-17T23:55:25.000312+0000","last_clean":"2024-03-17T23:55:25.000312+0000","last_became_active":"2024-03-17T23:54:55.854185+0000","last_became_peered":"2024-03-17T23:54:55.854185+0000","last_unstale":"2024-03-17T23:55:25.000312+0000","last_undegraded":"2024-03-17T23:55:25.000312+0000","last_fullsized":"2024-03-17T23:55:25.000312+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:42:13.424550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854342+0000","last_change":"2024-03-17T23:54:56.353311+0000","last_active":"2024-03-17T23:55:02.854342+0000","last_peered":"2024-03-17T23:55:02.854342+0000","last_clean":"2024-03-17T23:55:02.854342+0000","last_became_active":"2024-03-17T23:54:56.353013+0000","last_became_peered":"2024-03-17T23:54:56.353013+0000","last_unstale":"2024-03-17T23:55:02.854342+0000","last_undegraded":"2024-03-17T23:55:02.854342+0000","last_fullsized":"2024-03-17T23:55:02.854342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:21:20.119186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660812+0000","last_change":"2024-03-17T23:55:01.888595+0000","last_active":"2024-03-17T23:55:04.660812+0000","last_peered":"2024-03-17T23:55:04.660812+0000","last_clean":"2024-03-17T23:55:04.660812+0000","last_became_active":"2024-03-17T23:55:01.888154+0000","last_became_peered":"2024-03-17T23:55:01.888154+0000","last_unstale":"2024-03-17T23:55:04.660812+0000","last_undegraded":"2024-03-17T23:55:04.660812+0000","last_fullsized":"2024-03-17T23:55:04.660812+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:28:17.680458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:25.003941+0000","last_change":"2024-03-17T23:54:55.862756+0000","last_active":"2024-03-17T23:55:25.003941+0000","last_peered":"2024-03-17T23:55:25.003941+0000","last_clean":"2024-03-17T23:55:25.003941+0000","last_became_active":"2024-03-17T23:54:55.862298+0000","last_became_peered":"2024-03-17T23:54:55.862298+0000","last_unstale":"2024-03-17T23:55:25.003941+0000","last_undegraded":"2024-03-17T23:55:25.003941+0000","last_fullsized":"2024-03-17T23:55:25.003941+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:15:52.613136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.371626+0000","last_change":"2024-03-17T23:54:56.354359+0000","last_active":"2024-03-17T23:55:03.371626+0000","last_peered":"2024-03-17T23:55:03.371626+0000","last_clean":"2024-03-17T23:55:03.371626+0000","last_became_active":"2024-03-17T23:54:56.354002+0000","last_became_peered":"2024-03-17T23:54:56.354002+0000","last_unstale":"2024-03-17T23:55:03.371626+0000","last_undegraded":"2024-03-17T23:55:03.371626+0000","last_fullsized":"2024-03-17T23:55:03.371626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:41:12.128471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.664012+0000","last_change":"2024-03-17T23:55:01.880579+0000","last_active":"2024-03-17T23:55:04.664012+0000","last_peered":"2024-03-17T23:55:04.664012+0000","last_clean":"2024-03-17T23:55:04.664012+0000","last_became_active":"2024-03-17T23:55:01.879471+0000","last_became_peered":"2024-03-17T23:55:01.879471+0000","last_unstale":"2024-03-17T23:55:04.664012+0000","last_undegraded":"2024-03-17T23:55:04.664012+0000","last_fullsized":"2024-03-17T23:55:04.664012+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:51.117904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:26.011593+0000","last_change":"2024-03-17T23:54:55.842998+0000","last_active":"2024-03-17T23:55:26.011593+0000","last_peered":"2024-03-17T23:55:26.011593+0000","last_clean":"2024-03-17T23:55:26.011593+0000","last_became_active":"2024-03-17T23:54:55.842678+0000","last_became_peered":"2024-03-17T23:54:55.842678+0000","last_unstale":"2024-03-17T23:55:26.011593+0000","last_undegraded":"2024-03-17T23:55:26.011593+0000","last_fullsized":"2024-03-17T23:55:26.011593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:14.546738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.449645+0000","last_change":"2024-03-17T23:54:56.365881+0000","last_active":"2024-03-17T23:55:04.449645+0000","last_peered":"2024-03-17T23:55:04.449645+0000","last_clean":"2024-03-17T23:55:04.449645+0000","last_became_active":"2024-03-17T23:54:56.365560+0000","last_became_peered":"2024-03-17T23:54:56.365560+0000","last_unstale":"2024-03-17T23:55:04.449645+0000","last_undegraded":"2024-03-17T23:55:04.449645+0000","last_fullsized":"2024-03-17T23:55:04.449645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:38.436644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.371883+0000","last_change":"2024-03-17T23:55:01.882651+0000","last_active":"2024-03-17T23:55:03.371883+0000","last_peered":"2024-03-17T23:55:03.371883+0000","last_clean":"2024-03-17T23:55:03.371883+0000","last_became_active":"2024-03-17T23:55:01.881717+0000","last_became_peered":"2024-03-17T23:55:01.881717+0000","last_unstale":"2024-03-17T23:55:03.371883+0000","last_undegraded":"2024-03-17T23:55:03.371883+0000","last_fullsized":"2024-03-17T23:55:03.371883+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:21:08.644184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:23.987241+0000","last_change":"2024-03-17T23:54:55.854822+0000","last_active":"2024-03-17T23:55:23.987241+0000","last_peered":"2024-03-17T23:55:23.987241+0000","last_clean":"2024-03-17T23:55:23.987241+0000","last_became_active":"2024-03-17T23:54:55.854501+0000","last_became_peered":"2024-03-17T23:54:55.854501+0000","last_unstale":"2024-03-17T23:55:23.987241+0000","last_undegraded":"2024-03-17T23:55:23.987241+0000","last_fullsized":"2024-03-17T23:55:23.987241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:04:12.504798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853839+0000","last_change":"2024-03-17T23:54:56.361898+0000","last_active":"2024-03-17T23:55:02.853839+0000","last_peered":"2024-03-17T23:55:02.853839+0000","last_clean":"2024-03-17T23:55:02.853839+0000","last_became_active":"2024-03-17T23:54:56.361556+0000","last_became_peered":"2024-03-17T23:54:56.361556+0000","last_unstale":"2024-03-17T23:55:02.853839+0000","last_undegraded":"2024-03-17T23:55:02.853839+0000","last_fullsized":"2024-03-17T23:55:02.853839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:52:08.932801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.450498+0000","last_change":"2024-03-17T23:55:01.882628+0000","last_active":"2024-03-17T23:55:04.450498+0000","last_peered":"2024-03-17T23:55:04.450498+0000","last_clean":"2024-03-17T23:55:04.450498+0000","last_became_active":"2024-03-17T23:55:01.881886+0000","last_became_peered":"2024-03-17T23:55:01.881886+0000","last_unstale":"2024-03-17T23:55:04.450498+0000","last_undegraded":"2024-03-17T23:55:04.450498+0000","last_fullsized":"2024-03-17T23:55:04.450498+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:22:58.534442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.032189+0000","last_change":"2024-03-17T23:54:55.839925+0000","last_active":"2024-03-17T23:55:27.032189+0000","last_peered":"2024-03-17T23:55:27.032189+0000","last_clean":"2024-03-17T23:55:27.032189+0000","last_became_active":"2024-03-17T23:54:55.839626+0000","last_became_peered":"2024-03-17T23:54:55.839626+0000","last_unstale":"2024-03-17T23:55:27.032189+0000","last_undegraded":"2024-03-17T23:55:27.032189+0000","last_fullsized":"2024-03-17T23:55:27.032189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:39.456462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.661735+0000","last_change":"2024-03-17T23:54:56.352766+0000","last_active":"2024-03-17T23:55:04.661735+0000","last_peered":"2024-03-17T23:55:04.661735+0000","last_clean":"2024-03-17T23:55:04.661735+0000","last_became_active":"2024-03-17T23:54:56.351817+0000","last_became_peered":"2024-03-17T23:54:56.351817+0000","last_unstale":"2024-03-17T23:55:04.661735+0000","last_undegraded":"2024-03-17T23:55:04.661735+0000","last_fullsized":"2024-03-17T23:55:04.661735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:57:38.097163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222388+0000","last_change":"2024-03-17T23:55:01.892650+0000","last_active":"2024-03-17T23:55:03.222388+0000","last_peered":"2024-03-17T23:55:03.222388+0000","last_clean":"2024-03-17T23:55:03.222388+0000","last_became_active":"2024-03-17T23:55:01.892269+0000","last_became_peered":"2024-03-17T23:55:01.892269+0000","last_unstale":"2024-03-17T23:55:03.222388+0000","last_undegraded":"2024-03-17T23:55:03.222388+0000","last_fullsized":"2024-03-17T23:55:03.222388+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:36:17.525339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:23.988745+0000","last_change":"2024-03-17T23:54:55.863264+0000","last_active":"2024-03-17T23:55:23.988745+0000","last_peered":"2024-03-17T23:55:23.988745+0000","last_clean":"2024-03-17T23:55:23.988745+0000","last_became_active":"2024-03-17T23:54:55.863000+0000","last_became_peered":"2024-03-17T23:54:55.863000+0000","last_unstale":"2024-03-17T23:55:23.988745+0000","last_undegraded":"2024-03-17T23:55:23.988745+0000","last_fullsized":"2024-03-17T23:55:23.988745+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:58.208064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853795+0000","last_change":"2024-03-17T23:54:56.351161+0000","last_active":"2024-03-17T23:55:02.853795+0000","last_peered":"2024-03-17T23:55:02.853795+0000","last_clean":"2024-03-17T23:55:02.853795+0000","last_became_active":"2024-03-17T23:54:56.349787+0000","last_became_peered":"2024-03-17T23:54:56.349787+0000","last_unstale":"2024-03-17T23:55:02.853795+0000","last_undegraded":"2024-03-17T23:55:02.853795+0000","last_fullsized":"2024-03-17T23:55:02.853795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:19:28.635060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.451325+0000","last_change":"2024-03-17T23:55:01.882229+0000","last_active":"2024-03-17T23:55:04.451325+0000","last_peered":"2024-03-17T23:55:04.451325+0000","last_clean":"2024-03-17T23:55:04.451325+0000","last_became_active":"2024-03-17T23:55:01.881768+0000","last_became_peered":"2024-03-17T23:55:01.881768+0000","last_unstale":"2024-03-17T23:55:04.451325+0000","last_undegraded":"2024-03-17T23:55:04.451325+0000","last_fullsized":"2024-03-17T23:55:04.451325+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:03:35.149413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'12","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:23.986986+0000","last_change":"2024-03-17T23:54:55.846449+0000","last_active":"2024-03-17T23:55:23.986986+0000","last_peered":"2024-03-17T23:55:23.986986+0000","last_clean":"2024-03-17T23:55:23.986986+0000","last_became_active":"2024-03-17T23:54:55.846064+0000","last_became_peered":"2024-03-17T23:54:55.846064+0000","last_unstale":"2024-03-17T23:55:23.986986+0000","last_undegraded":"2024-03-17T23:55:23.986986+0000","last_fullsized":"2024-03-17T23:55:23.986986+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:45.985724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.178254+0000","last_change":"2024-03-17T23:54:56.343717+0000","last_active":"2024-03-17T23:55:03.178254+0000","last_peered":"2024-03-17T23:55:03.178254+0000","last_clean":"2024-03-17T23:55:03.178254+0000","last_became_active":"2024-03-17T23:54:56.343418+0000","last_became_peered":"2024-03-17T23:54:56.343418+0000","last_unstale":"2024-03-17T23:55:03.178254+0000","last_undegraded":"2024-03-17T23:55:03.178254+0000","last_fullsized":"2024-03-17T23:55:03.178254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:11:40.402878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854876+0000","last_change":"2024-03-17T23:55:01.884671+0000","last_active":"2024-03-17T23:55:02.854876+0000","last_peered":"2024-03-17T23:55:02.854876+0000","last_clean":"2024-03-17T23:55:02.854876+0000","last_became_active":"2024-03-17T23:55:01.884102+0000","last_became_peered":"2024-03-17T23:55:01.884102+0000","last_unstale":"2024-03-17T23:55:02.854876+0000","last_undegraded":"2024-03-17T23:55:02.854876+0000","last_fullsized":"2024-03-17T23:55:02.854876+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:53:31.198466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.039040+0000","last_change":"2024-03-17T23:54:55.862124+0000","last_active":"2024-03-17T23:55:28.039040+0000","last_peered":"2024-03-17T23:55:28.039040+0000","last_clean":"2024-03-17T23:55:28.039040+0000","last_became_active":"2024-03-17T23:54:55.861758+0000","last_became_peered":"2024-03-17T23:54:55.861758+0000","last_unstale":"2024-03-17T23:55:28.039040+0000","last_undegraded":"2024-03-17T23:55:28.039040+0000","last_fullsized":"2024-03-17T23:55:28.039040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:02:25.138388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.858015+0000","last_change":"2024-03-17T23:54:56.361699+0000","last_active":"2024-03-17T23:55:02.858015+0000","last_peered":"2024-03-17T23:55:02.858015+0000","last_clean":"2024-03-17T23:55:02.858015+0000","last_became_active":"2024-03-17T23:54:56.361365+0000","last_became_peered":"2024-03-17T23:54:56.361365+0000","last_unstale":"2024-03-17T23:55:02.858015+0000","last_undegraded":"2024-03-17T23:55:02.858015+0000","last_fullsized":"2024-03-17T23:55:02.858015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:24:30.649956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660140+0000","last_change":"2024-03-17T23:55:01.888676+0000","last_active":"2024-03-17T23:55:04.660140+0000","last_peered":"2024-03-17T23:55:04.660140+0000","last_clean":"2024-03-17T23:55:04.660140+0000","last_became_active":"2024-03-17T23:55:01.888224+0000","last_became_peered":"2024-03-17T23:55:01.888224+0000","last_unstale":"2024-03-17T23:55:04.660140+0000","last_undegraded":"2024-03-17T23:55:04.660140+0000","last_fullsized":"2024-03-17T23:55:04.660140+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:22.075096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:26.020005+0000","last_change":"2024-03-17T23:54:55.857947+0000","last_active":"2024-03-17T23:55:26.020005+0000","last_peered":"2024-03-17T23:55:26.020005+0000","last_clean":"2024-03-17T23:55:26.020005+0000","last_became_active":"2024-03-17T23:54:55.857595+0000","last_became_peered":"2024-03-17T23:54:55.857595+0000","last_unstale":"2024-03-17T23:55:26.020005+0000","last_undegraded":"2024-03-17T23:55:26.020005+0000","last_fullsized":"2024-03-17T23:55:26.020005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:25.433053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306866+0000","last_change":"2024-03-17T23:54:56.347220+0000","last_active":"2024-03-17T23:55:03.306866+0000","last_peered":"2024-03-17T23:55:03.306866+0000","last_clean":"2024-03-17T23:55:03.306866+0000","last_became_active":"2024-03-17T23:54:56.346600+0000","last_became_peered":"2024-03-17T23:54:56.346600+0000","last_unstale":"2024-03-17T23:55:03.306866+0000","last_undegraded":"2024-03-17T23:55:03.306866+0000","last_fullsized":"2024-03-17T23:55:03.306866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:38:25.575448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.452597+0000","last_change":"2024-03-17T23:55:01.887016+0000","last_active":"2024-03-17T23:55:04.452597+0000","last_peered":"2024-03-17T23:55:04.452597+0000","last_clean":"2024-03-17T23:55:04.452597+0000","last_became_active":"2024-03-17T23:55:01.886576+0000","last_became_peered":"2024-03-17T23:55:01.886576+0000","last_unstale":"2024-03-17T23:55:04.452597+0000","last_undegraded":"2024-03-17T23:55:04.452597+0000","last_fullsized":"2024-03-17T23:55:04.452597+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:41:26.995982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.045240+0000","last_change":"2024-03-17T23:54:55.840254+0000","last_active":"2024-03-17T23:55:28.045240+0000","last_peered":"2024-03-17T23:55:28.045240+0000","last_clean":"2024-03-17T23:55:28.045240+0000","last_became_active":"2024-03-17T23:54:55.839740+0000","last_became_peered":"2024-03-17T23:54:55.839740+0000","last_unstale":"2024-03-17T23:55:28.045240+0000","last_undegraded":"2024-03-17T23:55:28.045240+0000","last_fullsized":"2024-03-17T23:55:28.045240+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:39.664083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306815+0000","last_change":"2024-03-17T23:54:56.343262+0000","last_active":"2024-03-17T23:55:03.306815+0000","last_peered":"2024-03-17T23:55:03.306815+0000","last_clean":"2024-03-17T23:55:03.306815+0000","last_became_active":"2024-03-17T23:54:56.342924+0000","last_became_peered":"2024-03-17T23:54:56.342924+0000","last_unstale":"2024-03-17T23:55:03.306815+0000","last_undegraded":"2024-03-17T23:55:03.306815+0000","last_fullsized":"2024-03-17T23:55:03.306815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:35:19.074001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.860057+0000","last_change":"2024-03-17T23:55:01.885081+0000","last_active":"2024-03-17T23:55:02.860057+0000","last_peered":"2024-03-17T23:55:02.860057+0000","last_clean":"2024-03-17T23:55:02.860057+0000","last_became_active":"2024-03-17T23:55:01.883328+0000","last_became_peered":"2024-03-17T23:55:01.883328+0000","last_unstale":"2024-03-17T23:55:02.860057+0000","last_undegraded":"2024-03-17T23:55:02.860057+0000","last_fullsized":"2024-03-17T23:55:02.860057+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:12:10.206774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.032590+0000","last_change":"2024-03-17T23:54:55.852560+0000","last_active":"2024-03-17T23:55:27.032590+0000","last_peered":"2024-03-17T23:55:27.032590+0000","last_clean":"2024-03-17T23:55:27.032590+0000","last_became_active":"2024-03-17T23:54:55.852234+0000","last_became_peered":"2024-03-17T23:54:55.852234+0000","last_unstale":"2024-03-17T23:55:27.032590+0000","last_undegraded":"2024-03-17T23:55:27.032590+0000","last_fullsized":"2024-03-17T23:55:27.032590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:36:22.272022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853988+0000","last_change":"2024-03-17T23:54:56.345946+0000","last_active":"2024-03-17T23:55:02.853988+0000","last_peered":"2024-03-17T23:55:02.853988+0000","last_clean":"2024-03-17T23:55:02.853988+0000","last_became_active":"2024-03-17T23:54:56.345620+0000","last_became_peered":"2024-03-17T23:54:56.345620+0000","last_unstale":"2024-03-17T23:55:02.853988+0000","last_undegraded":"2024-03-17T23:55:02.853988+0000","last_fullsized":"2024-03-17T23:55:02.853988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:03:51.327966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854049+0000","last_change":"2024-03-17T23:55:01.884819+0000","last_active":"2024-03-17T23:55:02.854049+0000","last_peered":"2024-03-17T23:55:02.854049+0000","last_clean":"2024-03-17T23:55:02.854049+0000","last_became_active":"2024-03-17T23:55:01.884045+0000","last_became_peered":"2024-03-17T23:55:01.884045+0000","last_unstale":"2024-03-17T23:55:02.854049+0000","last_undegraded":"2024-03-17T23:55:02.854049+0000","last_fullsized":"2024-03-17T23:55:02.854049+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:10.147132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:26.991053+0000","last_change":"2024-03-17T23:54:55.840370+0000","last_active":"2024-03-17T23:55:26.991053+0000","last_peered":"2024-03-17T23:55:26.991053+0000","last_clean":"2024-03-17T23:55:26.991053+0000","last_became_active":"2024-03-17T23:54:55.840049+0000","last_became_peered":"2024-03-17T23:54:55.840049+0000","last_unstale":"2024-03-17T23:55:26.991053+0000","last_undegraded":"2024-03-17T23:55:26.991053+0000","last_fullsized":"2024-03-17T23:55:26.991053+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:40:46.523632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659909+0000","last_change":"2024-03-17T23:54:56.346533+0000","last_active":"2024-03-17T23:55:04.659909+0000","last_peered":"2024-03-17T23:55:04.659909+0000","last_clean":"2024-03-17T23:55:04.659909+0000","last_became_active":"2024-03-17T23:54:56.346214+0000","last_became_peered":"2024-03-17T23:54:56.346214+0000","last_unstale":"2024-03-17T23:55:04.659909+0000","last_undegraded":"2024-03-17T23:55:04.659909+0000","last_fullsized":"2024-03-17T23:55:04.659909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:06:59.017416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.178391+0000","last_change":"2024-03-17T23:55:02.318944+0000","last_active":"2024-03-17T23:55:03.178391+0000","last_peered":"2024-03-17T23:55:03.178391+0000","last_clean":"2024-03-17T23:55:03.178391+0000","last_became_active":"2024-03-17T23:55:02.318432+0000","last_became_peered":"2024-03-17T23:55:02.318432+0000","last_unstale":"2024-03-17T23:55:03.178391+0000","last_undegraded":"2024-03-17T23:55:03.178391+0000","last_fullsized":"2024-03-17T23:55:03.178391+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:23:47.986701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.452001+0000","last_change":"2024-03-17T23:54:55.873746+0000","last_active":"2024-03-17T23:55:04.452001+0000","last_peered":"2024-03-17T23:55:04.452001+0000","last_clean":"2024-03-17T23:55:04.452001+0000","last_became_active":"2024-03-17T23:54:55.872229+0000","last_became_peered":"2024-03-17T23:54:55.872229+0000","last_unstale":"2024-03-17T23:55:04.452001+0000","last_undegraded":"2024-03-17T23:55:04.452001+0000","last_fullsized":"2024-03-17T23:55:04.452001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:10.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659774+0000","last_change":"2024-03-17T23:54:56.361294+0000","last_active":"2024-03-17T23:55:04.659774+0000","last_peered":"2024-03-17T23:55:04.659774+0000","last_clean":"2024-03-17T23:55:04.659774+0000","last_became_active":"2024-03-17T23:54:56.360976+0000","last_became_peered":"2024-03-17T23:54:56.360976+0000","last_unstale":"2024-03-17T23:55:04.659774+0000","last_undegraded":"2024-03-17T23:55:04.659774+0000","last_fullsized":"2024-03-17T23:55:04.659774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:28:58.685991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.858765+0000","last_change":"2024-03-17T23:55:01.880221+0000","last_active":"2024-03-17T23:55:02.858765+0000","last_peered":"2024-03-17T23:55:02.858765+0000","last_clean":"2024-03-17T23:55:02.858765+0000","last_became_active":"2024-03-17T23:55:01.879785+0000","last_became_peered":"2024-03-17T23:55:01.879785+0000","last_unstale":"2024-03-17T23:55:02.858765+0000","last_undegraded":"2024-03-17T23:55:02.858765+0000","last_fullsized":"2024-03-17T23:55:02.858765+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:24:51.698857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.042458+0000","last_change":"2024-03-17T23:54:55.854176+0000","last_active":"2024-03-17T23:55:28.042458+0000","last_peered":"2024-03-17T23:55:28.042458+0000","last_clean":"2024-03-17T23:55:28.042458+0000","last_became_active":"2024-03-17T23:54:55.853859+0000","last_became_peered":"2024-03-17T23:54:55.853859+0000","last_unstale":"2024-03-17T23:55:28.042458+0000","last_undegraded":"2024-03-17T23:55:28.042458+0000","last_fullsized":"2024-03-17T23:55:28.042458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:43:43.319635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659587+0000","last_change":"2024-03-17T23:54:56.352800+0000","last_active":"2024-03-17T23:55:04.659587+0000","last_peered":"2024-03-17T23:55:04.659587+0000","last_clean":"2024-03-17T23:55:04.659587+0000","last_became_active":"2024-03-17T23:54:56.352466+0000","last_became_peered":"2024-03-17T23:54:56.352466+0000","last_unstale":"2024-03-17T23:55:04.659587+0000","last_undegraded":"2024-03-17T23:55:04.659587+0000","last_fullsized":"2024-03-17T23:55:04.659587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:33:34.895301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.859987+0000","last_change":"2024-03-17T23:55:02.320494+0000","last_active":"2024-03-17T23:55:02.859987+0000","last_peered":"2024-03-17T23:55:02.859987+0000","last_clean":"2024-03-17T23:55:02.859987+0000","last_became_active":"2024-03-17T23:55:02.319188+0000","last_became_peered":"2024-03-17T23:55:02.319188+0000","last_unstale":"2024-03-17T23:55:02.859987+0000","last_undegraded":"2024-03-17T23:55:02.859987+0000","last_fullsized":"2024-03-17T23:55:02.859987+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:30:06.348718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854467+0000","last_change":"2024-03-17T23:54:56.340798+0000","last_active":"2024-03-17T23:55:02.854467+0000","last_peered":"2024-03-17T23:55:02.854467+0000","last_clean":"2024-03-17T23:55:02.854467+0000","last_became_active":"2024-03-17T23:54:56.340386+0000","last_became_peered":"2024-03-17T23:54:56.340386+0000","last_unstale":"2024-03-17T23:55:02.854467+0000","last_undegraded":"2024-03-17T23:55:02.854467+0000","last_fullsized":"2024-03-17T23:55:02.854467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:57:58.546464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:25.003386+0000","last_change":"2024-03-17T23:54:55.870475+0000","last_active":"2024-03-17T23:55:25.003386+0000","last_peered":"2024-03-17T23:55:25.003386+0000","last_clean":"2024-03-17T23:55:25.003386+0000","last_became_active":"2024-03-17T23:54:55.870119+0000","last_became_peered":"2024-03-17T23:54:55.870119+0000","last_unstale":"2024-03-17T23:55:25.003386+0000","last_undegraded":"2024-03-17T23:55:25.003386+0000","last_fullsized":"2024-03-17T23:55:25.003386+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:12:15.421117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.032609+0000","last_change":"2024-03-17T23:55:01.892759+0000","last_active":"2024-03-17T23:55:04.032609+0000","last_peered":"2024-03-17T23:55:04.032609+0000","last_clean":"2024-03-17T23:55:04.032609+0000","last_became_active":"2024-03-17T23:55:01.892240+0000","last_became_peered":"2024-03-17T23:55:01.892240+0000","last_unstale":"2024-03-17T23:55:04.032609+0000","last_undegraded":"2024-03-17T23:55:04.032609+0000","last_fullsized":"2024-03-17T23:55:04.032609+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:16:24.391971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.371961+0000","last_change":"2024-03-17T23:54:56.354898+0000","last_active":"2024-03-17T23:55:03.371961+0000","last_peered":"2024-03-17T23:55:03.371961+0000","last_clean":"2024-03-17T23:55:03.371961+0000","last_became_active":"2024-03-17T23:54:56.354345+0000","last_became_peered":"2024-03-17T23:54:56.354345+0000","last_unstale":"2024-03-17T23:55:03.371961+0000","last_undegraded":"2024-03-17T23:55:03.371961+0000","last_fullsized":"2024-03-17T23:55:03.371961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:18:56.962412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.038111+0000","last_change":"2024-03-17T23:54:55.846706+0000","last_active":"2024-03-17T23:55:28.038111+0000","last_peered":"2024-03-17T23:55:28.038111+0000","last_clean":"2024-03-17T23:55:28.038111+0000","last_became_active":"2024-03-17T23:54:55.846372+0000","last_became_peered":"2024-03-17T23:54:55.846372+0000","last_unstale":"2024-03-17T23:55:28.038111+0000","last_undegraded":"2024-03-17T23:55:28.038111+0000","last_fullsized":"2024-03-17T23:55:28.038111+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:28:47.697975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306579+0000","last_change":"2024-03-17T23:53:34.383413+0000","last_active":"2024-03-17T23:55:03.306579+0000","last_peered":"2024-03-17T23:55:03.306579+0000","last_clean":"2024-03-17T23:55:03.306579+0000","last_became_active":"2024-03-17T23:53:34.382875+0000","last_became_peered":"2024-03-17T23:53:34.382875+0000","last_unstale":"2024-03-17T23:55:03.306579+0000","last_undegraded":"2024-03-17T23:55:03.306579+0000","last_fullsized":"2024-03-17T23:55:03.306579+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:38.641018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:38.641018+0000","last_clean_scrub_stamp":"2024-03-17T23:49:38.641018+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:49:45.234628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854345+0000","last_change":"2024-03-17T23:55:02.319161+0000","last_active":"2024-03-17T23:55:02.854345+0000","last_peered":"2024-03-17T23:55:02.854345+0000","last_clean":"2024-03-17T23:55:02.854345+0000","last_became_active":"2024-03-17T23:55:02.318561+0000","last_became_peered":"2024-03-17T23:55:02.318561+0000","last_unstale":"2024-03-17T23:55:02.854345+0000","last_undegraded":"2024-03-17T23:55:02.854345+0000","last_fullsized":"2024-03-17T23:55:02.854345+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:10.585085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.857455+0000","last_change":"2024-03-17T23:54:56.346097+0000","last_active":"2024-03-17T23:55:02.857455+0000","last_peered":"2024-03-17T23:55:02.857455+0000","last_clean":"2024-03-17T23:55:02.857455+0000","last_became_active":"2024-03-17T23:54:56.345762+0000","last_became_peered":"2024-03-17T23:54:56.345762+0000","last_unstale":"2024-03-17T23:55:02.857455+0000","last_undegraded":"2024-03-17T23:55:02.857455+0000","last_fullsized":"2024-03-17T23:55:02.857455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:43:15.860897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:24.996120+0000","last_change":"2024-03-17T23:54:55.840743+0000","last_active":"2024-03-17T23:55:24.996120+0000","last_peered":"2024-03-17T23:55:24.996120+0000","last_clean":"2024-03-17T23:55:24.996120+0000","last_became_active":"2024-03-17T23:54:55.840463+0000","last_became_peered":"2024-03-17T23:54:55.840463+0000","last_unstale":"2024-03-17T23:55:24.996120+0000","last_undegraded":"2024-03-17T23:55:24.996120+0000","last_fullsized":"2024-03-17T23:55:24.996120+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:24:24.130911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660318+0000","last_change":"2024-03-17T23:55:01.876895+0000","last_active":"2024-03-17T23:55:04.660318+0000","last_peered":"2024-03-17T23:55:04.660318+0000","last_clean":"2024-03-17T23:55:04.660318+0000","last_became_active":"2024-03-17T23:55:01.876443+0000","last_became_peered":"2024-03-17T23:55:01.876443+0000","last_unstale":"2024-03-17T23:55:04.660318+0000","last_undegraded":"2024-03-17T23:55:04.660318+0000","last_fullsized":"2024-03-17T23:55:04.660318+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:51:55.737167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.034886+0000","last_change":"2024-03-17T23:54:55.863601+0000","last_active":"2024-03-17T23:55:27.034886+0000","last_peered":"2024-03-17T23:55:27.034886+0000","last_clean":"2024-03-17T23:55:27.034886+0000","last_became_active":"2024-03-17T23:54:55.863247+0000","last_became_peered":"2024-03-17T23:54:55.863247+0000","last_unstale":"2024-03-17T23:55:27.034886+0000","last_undegraded":"2024-03-17T23:55:27.034886+0000","last_fullsized":"2024-03-17T23:55:27.034886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:53:27.747199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.859188+0000","last_change":"2024-03-17T23:54:56.365913+0000","last_active":"2024-03-17T23:55:02.859188+0000","last_peered":"2024-03-17T23:55:02.859188+0000","last_clean":"2024-03-17T23:55:02.859188+0000","last_became_active":"2024-03-17T23:54:56.365578+0000","last_became_peered":"2024-03-17T23:54:56.365578+0000","last_unstale":"2024-03-17T23:55:02.859188+0000","last_undegraded":"2024-03-17T23:55:02.859188+0000","last_fullsized":"2024-03-17T23:55:02.859188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:49.330026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.371675+0000","last_change":"2024-03-17T23:55:02.319060+0000","last_active":"2024-03-17T23:55:03.371675+0000","last_peered":"2024-03-17T23:55:03.371675+0000","last_clean":"2024-03-17T23:55:03.371675+0000","last_became_active":"2024-03-17T23:55:02.318542+0000","last_became_peered":"2024-03-17T23:55:02.318542+0000","last_unstale":"2024-03-17T23:55:03.371675+0000","last_undegraded":"2024-03-17T23:55:03.371675+0000","last_fullsized":"2024-03-17T23:55:03.371675+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:59:27.334039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'9","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.318574+0000","last_change":"2024-03-17T23:54:55.861011+0000","last_active":"2024-03-17T23:55:27.318574+0000","last_peered":"2024-03-17T23:55:27.318574+0000","last_clean":"2024-03-17T23:55:27.318574+0000","last_became_active":"2024-03-17T23:54:55.860388+0000","last_became_peered":"2024-03-17T23:54:55.860388+0000","last_unstale":"2024-03-17T23:55:27.318574+0000","last_undegraded":"2024-03-17T23:55:27.318574+0000","last_fullsized":"2024-03-17T23:55:27.318574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:20:11.943034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.372867+0000","last_change":"2024-03-17T23:54:56.357572+0000","last_active":"2024-03-17T23:55:03.372867+0000","last_peered":"2024-03-17T23:55:03.372867+0000","last_clean":"2024-03-17T23:55:03.372867+0000","last_became_active":"2024-03-17T23:54:56.357226+0000","last_became_peered":"2024-03-17T23:54:56.357226+0000","last_unstale":"2024-03-17T23:55:03.372867+0000","last_undegraded":"2024-03-17T23:55:03.372867+0000","last_fullsized":"2024-03-17T23:55:03.372867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:35:44.939454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207294+0000","last_change":"2024-03-17T23:55:01.860239+0000","last_active":"2024-03-17T23:55:04.207294+0000","last_peered":"2024-03-17T23:55:04.207294+0000","last_clean":"2024-03-17T23:55:04.207294+0000","last_became_active":"2024-03-17T23:55:01.859754+0000","last_became_peered":"2024-03-17T23:55:01.859754+0000","last_unstale":"2024-03-17T23:55:04.207294+0000","last_undegraded":"2024-03-17T23:55:04.207294+0000","last_fullsized":"2024-03-17T23:55:04.207294+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:30:16.833040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.034726+0000","last_change":"2024-03-17T23:54:55.855917+0000","last_active":"2024-03-17T23:55:27.034726+0000","last_peered":"2024-03-17T23:55:27.034726+0000","last_clean":"2024-03-17T23:55:27.034726+0000","last_became_active":"2024-03-17T23:54:55.855562+0000","last_became_peered":"2024-03-17T23:54:55.855562+0000","last_unstale":"2024-03-17T23:55:27.034726+0000","last_undegraded":"2024-03-17T23:55:27.034726+0000","last_fullsized":"2024-03-17T23:55:27.034726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:44.928086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660707+0000","last_change":"2024-03-17T23:54:56.362634+0000","last_active":"2024-03-17T23:55:04.660707+0000","last_peered":"2024-03-17T23:55:04.660707+0000","last_clean":"2024-03-17T23:55:04.660707+0000","last_became_active":"2024-03-17T23:54:56.361910+0000","last_became_peered":"2024-03-17T23:54:56.361910+0000","last_unstale":"2024-03-17T23:55:04.660707+0000","last_undegraded":"2024-03-17T23:55:04.660707+0000","last_fullsized":"2024-03-17T23:55:04.660707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:20:34.988074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222412+0000","last_change":"2024-03-17T23:55:02.319090+0000","last_active":"2024-03-17T23:55:03.222412+0000","last_peered":"2024-03-17T23:55:03.222412+0000","last_clean":"2024-03-17T23:55:03.222412+0000","last_became_active":"2024-03-17T23:55:02.318695+0000","last_became_peered":"2024-03-17T23:55:02.318695+0000","last_unstale":"2024-03-17T23:55:03.222412+0000","last_undegraded":"2024-03-17T23:55:03.222412+0000","last_fullsized":"2024-03-17T23:55:03.222412+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:32:13.984797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:23.988273+0000","last_change":"2024-03-17T23:54:55.852999+0000","last_active":"2024-03-17T23:55:23.988273+0000","last_peered":"2024-03-17T23:55:23.988273+0000","last_clean":"2024-03-17T23:55:23.988273+0000","last_became_active":"2024-03-17T23:54:55.852653+0000","last_became_peered":"2024-03-17T23:54:55.852653+0000","last_unstale":"2024-03-17T23:55:23.988273+0000","last_undegraded":"2024-03-17T23:55:23.988273+0000","last_fullsized":"2024-03-17T23:55:23.988273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:04:10.561807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.177862+0000","last_change":"2024-03-17T23:54:56.345171+0000","last_active":"2024-03-17T23:55:03.177862+0000","last_peered":"2024-03-17T23:55:03.177862+0000","last_clean":"2024-03-17T23:55:03.177862+0000","last_became_active":"2024-03-17T23:54:56.344146+0000","last_became_peered":"2024-03-17T23:54:56.344146+0000","last_unstale":"2024-03-17T23:55:03.177862+0000","last_undegraded":"2024-03-17T23:55:03.177862+0000","last_fullsized":"2024-03-17T23:55:03.177862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:08.546622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.452941+0000","last_change":"2024-03-17T23:55:01.882655+0000","last_active":"2024-03-17T23:55:04.452941+0000","last_peered":"2024-03-17T23:55:04.452941+0000","last_clean":"2024-03-17T23:55:04.452941+0000","last_became_active":"2024-03-17T23:55:01.882147+0000","last_became_peered":"2024-03-17T23:55:01.882147+0000","last_unstale":"2024-03-17T23:55:04.452941+0000","last_undegraded":"2024-03-17T23:55:04.452941+0000","last_fullsized":"2024-03-17T23:55:04.452941+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:17:55.653103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:26.023625+0000","last_change":"2024-03-17T23:54:55.862499+0000","last_active":"2024-03-17T23:55:26.023625+0000","last_peered":"2024-03-17T23:55:26.023625+0000","last_clean":"2024-03-17T23:55:26.023625+0000","last_became_active":"2024-03-17T23:54:55.862003+0000","last_became_peered":"2024-03-17T23:54:55.862003+0000","last_unstale":"2024-03-17T23:55:26.023625+0000","last_undegraded":"2024-03-17T23:55:26.023625+0000","last_fullsized":"2024-03-17T23:55:26.023625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:37:19.046543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659875+0000","last_change":"2024-03-17T23:54:56.352154+0000","last_active":"2024-03-17T23:55:04.659875+0000","last_peered":"2024-03-17T23:55:04.659875+0000","last_clean":"2024-03-17T23:55:04.659875+0000","last_became_active":"2024-03-17T23:54:56.351818+0000","last_became_peered":"2024-03-17T23:54:56.351818+0000","last_unstale":"2024-03-17T23:55:04.659875+0000","last_undegraded":"2024-03-17T23:55:04.659875+0000","last_fullsized":"2024-03-17T23:55:04.659875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:44:46.538778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.338498+0000","last_change":"2024-03-17T23:55:01.876605+0000","last_active":"2024-03-17T23:55:04.338498+0000","last_peered":"2024-03-17T23:55:04.338498+0000","last_clean":"2024-03-17T23:55:04.338498+0000","last_became_active":"2024-03-17T23:55:01.876253+0000","last_became_peered":"2024-03-17T23:55:01.876253+0000","last_unstale":"2024-03-17T23:55:04.338498+0000","last_undegraded":"2024-03-17T23:55:04.338498+0000","last_fullsized":"2024-03-17T23:55:04.338498+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:27:43.271798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:26.019930+0000","last_change":"2024-03-17T23:54:55.862209+0000","last_active":"2024-03-17T23:55:26.019930+0000","last_peered":"2024-03-17T23:55:26.019930+0000","last_clean":"2024-03-17T23:55:26.019930+0000","last_became_active":"2024-03-17T23:54:55.861829+0000","last_became_peered":"2024-03-17T23:54:55.861829+0000","last_unstale":"2024-03-17T23:55:26.019930+0000","last_undegraded":"2024-03-17T23:55:26.019930+0000","last_fullsized":"2024-03-17T23:55:26.019930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:13:58.756538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.372408+0000","last_change":"2024-03-17T23:54:56.366869+0000","last_active":"2024-03-17T23:55:03.372408+0000","last_peered":"2024-03-17T23:55:03.372408+0000","last_clean":"2024-03-17T23:55:03.372408+0000","last_became_active":"2024-03-17T23:54:56.365380+0000","last_became_peered":"2024-03-17T23:54:56.365380+0000","last_unstale":"2024-03-17T23:55:03.372408+0000","last_undegraded":"2024-03-17T23:55:03.372408+0000","last_fullsized":"2024-03-17T23:55:03.372408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:28:03.010849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.222441+0000","last_change":"2024-03-17T23:55:02.318820+0000","last_active":"2024-03-17T23:55:03.222441+0000","last_peered":"2024-03-17T23:55:03.222441+0000","last_clean":"2024-03-17T23:55:03.222441+0000","last_became_active":"2024-03-17T23:55:02.318308+0000","last_became_peered":"2024-03-17T23:55:02.318308+0000","last_unstale":"2024-03-17T23:55:03.222441+0000","last_undegraded":"2024-03-17T23:55:03.222441+0000","last_fullsized":"2024-03-17T23:55:03.222441+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:40:35.224105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:23.987107+0000","last_change":"2024-03-17T23:54:55.837980+0000","last_active":"2024-03-17T23:55:23.987107+0000","last_peered":"2024-03-17T23:55:23.987107+0000","last_clean":"2024-03-17T23:55:23.987107+0000","last_became_active":"2024-03-17T23:54:55.837614+0000","last_became_peered":"2024-03-17T23:54:55.837614+0000","last_unstale":"2024-03-17T23:55:23.987107+0000","last_undegraded":"2024-03-17T23:55:23.987107+0000","last_fullsized":"2024-03-17T23:55:23.987107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:05:07.228281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306819+0000","last_change":"2024-03-17T23:54:56.348534+0000","last_active":"2024-03-17T23:55:03.306819+0000","last_peered":"2024-03-17T23:55:03.306819+0000","last_clean":"2024-03-17T23:55:03.306819+0000","last_became_active":"2024-03-17T23:54:56.348138+0000","last_became_peered":"2024-03-17T23:54:56.348138+0000","last_unstale":"2024-03-17T23:55:03.306819+0000","last_undegraded":"2024-03-17T23:55:03.306819+0000","last_fullsized":"2024-03-17T23:55:03.306819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:47:49.481608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853635+0000","last_change":"2024-03-17T23:55:01.876287+0000","last_active":"2024-03-17T23:55:02.853635+0000","last_peered":"2024-03-17T23:55:02.853635+0000","last_clean":"2024-03-17T23:55:02.853635+0000","last_became_active":"2024-03-17T23:55:01.875861+0000","last_became_peered":"2024-03-17T23:55:01.875861+0000","last_unstale":"2024-03-17T23:55:02.853635+0000","last_undegraded":"2024-03-17T23:55:02.853635+0000","last_fullsized":"2024-03-17T23:55:02.853635+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:33:59.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:28.042437+0000","last_change":"2024-03-17T23:54:55.870989+0000","last_active":"2024-03-17T23:55:28.042437+0000","last_peered":"2024-03-17T23:55:28.042437+0000","last_clean":"2024-03-17T23:55:28.042437+0000","last_became_active":"2024-03-17T23:54:55.870633+0000","last_became_peered":"2024-03-17T23:54:55.870633+0000","last_unstale":"2024-03-17T23:55:28.042437+0000","last_undegraded":"2024-03-17T23:55:28.042437+0000","last_fullsized":"2024-03-17T23:55:28.042437+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:55.310289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.307089+0000","last_change":"2024-03-17T23:54:56.345874+0000","last_active":"2024-03-17T23:55:03.307089+0000","last_peered":"2024-03-17T23:55:03.307089+0000","last_clean":"2024-03-17T23:55:03.307089+0000","last_became_active":"2024-03-17T23:54:56.345547+0000","last_became_peered":"2024-03-17T23:54:56.345547+0000","last_unstale":"2024-03-17T23:55:03.307089+0000","last_undegraded":"2024-03-17T23:55:03.307089+0000","last_fullsized":"2024-03-17T23:55:03.307089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:52:31.724224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.663521+0000","last_change":"2024-03-17T23:55:01.892947+0000","last_active":"2024-03-17T23:55:04.663521+0000","last_peered":"2024-03-17T23:55:04.663521+0000","last_clean":"2024-03-17T23:55:04.663521+0000","last_became_active":"2024-03-17T23:55:01.892481+0000","last_became_peered":"2024-03-17T23:55:01.892481+0000","last_unstale":"2024-03-17T23:55:04.663521+0000","last_undegraded":"2024-03-17T23:55:04.663521+0000","last_fullsized":"2024-03-17T23:55:04.663521+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:07:14.901490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:26.019965+0000","last_change":"2024-03-17T23:54:55.857943+0000","last_active":"2024-03-17T23:55:26.019965+0000","last_peered":"2024-03-17T23:55:26.019965+0000","last_clean":"2024-03-17T23:55:26.019965+0000","last_became_active":"2024-03-17T23:54:55.857592+0000","last_became_peered":"2024-03-17T23:54:55.857592+0000","last_unstale":"2024-03-17T23:55:26.019965+0000","last_undegraded":"2024-03-17T23:55:26.019965+0000","last_fullsized":"2024-03-17T23:55:26.019965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:50:03.643518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.306854+0000","last_change":"2024-03-17T23:54:56.357496+0000","last_active":"2024-03-17T23:55:03.306854+0000","last_peered":"2024-03-17T23:55:03.306854+0000","last_clean":"2024-03-17T23:55:03.306854+0000","last_became_active":"2024-03-17T23:54:56.357242+0000","last_became_peered":"2024-03-17T23:54:56.357242+0000","last_unstale":"2024-03-17T23:55:03.306854+0000","last_undegraded":"2024-03-17T23:55:03.306854+0000","last_fullsized":"2024-03-17T23:55:03.306854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:52:49.941623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853798+0000","last_change":"2024-03-17T23:55:01.866288+0000","last_active":"2024-03-17T23:55:02.853798+0000","last_peered":"2024-03-17T23:55:02.853798+0000","last_clean":"2024-03-17T23:55:02.853798+0000","last_became_active":"2024-03-17T23:55:01.865875+0000","last_became_peered":"2024-03-17T23:55:01.865875+0000","last_unstale":"2024-03-17T23:55:02.853798+0000","last_undegraded":"2024-03-17T23:55:02.853798+0000","last_fullsized":"2024-03-17T23:55:02.853798+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:45:43.198830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:27.036707+0000","last_change":"2024-03-17T23:54:55.871971+0000","last_active":"2024-03-17T23:55:27.036707+0000","last_peered":"2024-03-17T23:55:27.036707+0000","last_clean":"2024-03-17T23:55:27.036707+0000","last_became_active":"2024-03-17T23:54:55.871434+0000","last_became_peered":"2024-03-17T23:54:55.871434+0000","last_unstale":"2024-03-17T23:55:27.036707+0000","last_undegraded":"2024-03-17T23:55:27.036707+0000","last_fullsized":"2024-03-17T23:55:27.036707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:41:10.667628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.307065+0000","last_change":"2024-03-17T23:54:56.343595+0000","last_active":"2024-03-17T23:55:03.307065+0000","last_peered":"2024-03-17T23:55:03.307065+0000","last_clean":"2024-03-17T23:55:03.307065+0000","last_became_active":"2024-03-17T23:54:56.343270+0000","last_became_peered":"2024-03-17T23:54:56.343270+0000","last_unstale":"2024-03-17T23:55:03.307065+0000","last_undegraded":"2024-03-17T23:55:03.307065+0000","last_fullsized":"2024-03-17T23:55:03.307065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:22:05.904023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659693+0000","last_change":"2024-03-17T23:54:56.346571+0000","last_active":"2024-03-17T23:55:04.659693+0000","last_peered":"2024-03-17T23:55:04.659693+0000","last_clean":"2024-03-17T23:55:04.659693+0000","last_became_active":"2024-03-17T23:54:56.346234+0000","last_became_peered":"2024-03-17T23:54:56.346234+0000","last_unstale":"2024-03-17T23:55:04.659693+0000","last_undegraded":"2024-03-17T23:55:04.659693+0000","last_fullsized":"2024-03-17T23:55:04.659693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:25.448506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207633+0000","last_change":"2024-03-17T23:54:56.342299+0000","last_active":"2024-03-17T23:55:04.207633+0000","last_peered":"2024-03-17T23:55:04.207633+0000","last_clean":"2024-03-17T23:55:04.207633+0000","last_became_active":"2024-03-17T23:54:56.341988+0000","last_became_peered":"2024-03-17T23:54:56.341988+0000","last_unstale":"2024-03-17T23:55:04.207633+0000","last_undegraded":"2024-03-17T23:55:04.207633+0000","last_fullsized":"2024-03-17T23:55:04.207633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:03:38.595595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207300+0000","last_change":"2024-03-17T23:54:56.360601+0000","last_active":"2024-03-17T23:55:04.207300+0000","last_peered":"2024-03-17T23:55:04.207300+0000","last_clean":"2024-03-17T23:55:04.207300+0000","last_became_active":"2024-03-17T23:54:56.360215+0000","last_became_peered":"2024-03-17T23:54:56.360215+0000","last_unstale":"2024-03-17T23:55:04.207300+0000","last_undegraded":"2024-03-17T23:55:04.207300+0000","last_fullsized":"2024-03-17T23:55:04.207300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:42:07.661460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.660646+0000","last_change":"2024-03-17T23:54:56.358523+0000","last_active":"2024-03-17T23:55:04.660646+0000","last_peered":"2024-03-17T23:55:04.660646+0000","last_clean":"2024-03-17T23:55:04.660646+0000","last_became_active":"2024-03-17T23:54:56.358172+0000","last_became_peered":"2024-03-17T23:54:56.358172+0000","last_unstale":"2024-03-17T23:55:04.660646+0000","last_undegraded":"2024-03-17T23:55:04.660646+0000","last_fullsized":"2024-03-17T23:55:04.660646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:00.542176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.859048+0000","last_change":"2024-03-17T23:54:56.348619+0000","last_active":"2024-03-17T23:55:02.859048+0000","last_peered":"2024-03-17T23:55:02.859048+0000","last_clean":"2024-03-17T23:55:02.859048+0000","last_became_active":"2024-03-17T23:54:56.348309+0000","last_became_peered":"2024-03-17T23:54:56.348309+0000","last_unstale":"2024-03-17T23:55:02.859048+0000","last_undegraded":"2024-03-17T23:55:02.859048+0000","last_fullsized":"2024-03-17T23:55:02.859048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:38:42.130689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659754+0000","last_change":"2024-03-17T23:54:56.347736+0000","last_active":"2024-03-17T23:55:04.659754+0000","last_peered":"2024-03-17T23:55:04.659754+0000","last_clean":"2024-03-17T23:55:04.659754+0000","last_became_active":"2024-03-17T23:54:56.347283+0000","last_became_peered":"2024-03-17T23:54:56.347283+0000","last_unstale":"2024-03-17T23:55:04.659754+0000","last_undegraded":"2024-03-17T23:55:04.659754+0000","last_fullsized":"2024-03-17T23:55:04.659754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:35:00.683852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.207820+0000","last_change":"2024-03-17T23:54:56.344574+0000","last_active":"2024-03-17T23:55:04.207820+0000","last_peered":"2024-03-17T23:55:04.207820+0000","last_clean":"2024-03-17T23:55:04.207820+0000","last_became_active":"2024-03-17T23:54:56.344272+0000","last_became_peered":"2024-03-17T23:54:56.344272+0000","last_unstale":"2024-03-17T23:55:04.207820+0000","last_undegraded":"2024-03-17T23:55:04.207820+0000","last_fullsized":"2024-03-17T23:55:04.207820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:37:37.134152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.372977+0000","last_change":"2024-03-17T23:54:56.365633+0000","last_active":"2024-03-17T23:55:03.372977+0000","last_peered":"2024-03-17T23:55:03.372977+0000","last_clean":"2024-03-17T23:55:03.372977+0000","last_became_active":"2024-03-17T23:54:56.364533+0000","last_became_peered":"2024-03-17T23:54:56.364533+0000","last_unstale":"2024-03-17T23:55:03.372977+0000","last_undegraded":"2024-03-17T23:55:03.372977+0000","last_fullsized":"2024-03-17T23:55:03.372977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:05:20.848672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.854152+0000","last_change":"2024-03-17T23:54:56.365614+0000","last_active":"2024-03-17T23:55:02.854152+0000","last_peered":"2024-03-17T23:55:02.854152+0000","last_clean":"2024-03-17T23:55:02.854152+0000","last_became_active":"2024-03-17T23:54:56.365301+0000","last_became_peered":"2024-03-17T23:54:56.365301+0000","last_unstale":"2024-03-17T23:55:02.854152+0000","last_undegraded":"2024-03-17T23:55:02.854152+0000","last_fullsized":"2024-03-17T23:55:02.854152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:20.122964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:03.371692+0000","last_change":"2024-03-17T23:54:56.365531+0000","last_active":"2024-03-17T23:55:03.371692+0000","last_peered":"2024-03-17T23:55:03.371692+0000","last_clean":"2024-03-17T23:55:03.371692+0000","last_became_active":"2024-03-17T23:54:56.364675+0000","last_became_peered":"2024-03-17T23:54:56.364675+0000","last_unstale":"2024-03-17T23:55:03.371692+0000","last_undegraded":"2024-03-17T23:55:03.371692+0000","last_fullsized":"2024-03-17T23:55:03.371692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:25:49.612055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.853681+0000","last_change":"2024-03-17T23:54:56.355306+0000","last_active":"2024-03-17T23:55:02.853681+0000","last_peered":"2024-03-17T23:55:02.853681+0000","last_clean":"2024-03-17T23:55:02.853681+0000","last_became_active":"2024-03-17T23:54:56.354975+0000","last_became_peered":"2024-03-17T23:54:56.354975+0000","last_unstale":"2024-03-17T23:55:02.853681+0000","last_undegraded":"2024-03-17T23:55:02.853681+0000","last_fullsized":"2024-03-17T23:55:02.853681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:58.606691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:02.858602+0000","last_change":"2024-03-17T23:54:56.350333+0000","last_active":"2024-03-17T23:55:02.858602+0000","last_peered":"2024-03-17T23:55:02.858602+0000","last_clean":"2024-03-17T23:55:02.858602+0000","last_became_active":"2024-03-17T23:54:56.349953+0000","last_became_peered":"2024-03-17T23:54:56.349953+0000","last_unstale":"2024-03-17T23:55:02.858602+0000","last_undegraded":"2024-03-17T23:55:02.858602+0000","last_fullsized":"2024-03-17T23:55:02.858602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:19:34.639852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-17T23:55:04.659730+0000","last_change":"2024-03-17T23:54:56.343143+0000","last_active":"2024-03-17T23:55:04.659730+0000","last_peered":"2024-03-17T23:55:04.659730+0000","last_clean":"2024-03-17T23:55:04.659730+0000","last_became_active":"2024-03-17T23:54:56.342772+0000","last_became_peered":"2024-03-17T23:54:56.342772+0000","last_unstale":"2024-03-17T23:55:04.659730+0000","last_undegraded":"2024-03-17T23:55:04.659730+0000","last_fullsized":"2024-03-17T23:55:04.659730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:33:45.668975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246647,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10565,"num_objects":93,"num_object_clones":0,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":201,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":107913,"data_compressed":3670,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":490682,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645336,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1081182,"data_compressed":50389,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808863,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":494708,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070501,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":490990,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332138,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":1087487,"data_compressed":50389,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":486116,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":529052,"data_compressed":16519,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":487572,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":529663,"data_compressed":16519,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":485802,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":490190,"data_compressed":14684,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":1083069,"data_compressed":50389,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34061,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34061,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":43658,"data_compressed":1835,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":44269,"data_compressed":1835,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:55:30.825 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-17T23:55:30.826 INFO:tasks.cephadm:Setting up client nodes... 2024-03-17T23:55:30.826 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-17T23:55:31.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:31 smithi012 ceph-mon[31541]: pgmap v338: 129 pgs: 129 active+clean; 587 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4.4 KiB/s wr, 15 op/s 2024-03-17T23:55:31.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:31 smithi012 ceph-mon[31541]: from='client.24872 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:55:31.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:31 smithi012 ceph-mon[31541]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:55:31.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:31 smithi191 ceph-mon[36708]: pgmap v338: 129 pgs: 129 active+clean; 587 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4.4 KiB/s wr, 15 op/s 2024-03-17T23:55:31.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:31 smithi191 ceph-mon[36708]: from='client.24872 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:55:31.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:31 smithi191 ceph-mon[36708]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:55:31.369 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:31.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:31 smithi079 ceph-mon[36241]: pgmap v338: 129 pgs: 129 active+clean; 587 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4.4 KiB/s wr, 15 op/s 2024-03-17T23:55:31.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:31 smithi079 ceph-mon[36241]: from='client.24872 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:55:31.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:31 smithi079 ceph-mon[36241]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:55:33.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:33 smithi012 ceph-mon[31541]: pgmap v339: 129 pgs: 129 active+clean; 629 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 8.5 KiB/s wr, 18 op/s 2024-03-17T23:55:33.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:33 smithi079 ceph-mon[36241]: pgmap v339: 129 pgs: 129 active+clean; 629 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 8.5 KiB/s wr, 18 op/s 2024-03-17T23:55:33.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:33 smithi191 ceph-mon[36708]: pgmap v339: 129 pgs: 129 active+clean; 629 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 8.5 KiB/s wr, 18 op/s 2024-03-17T23:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.743+0000 7f3b52000700 1 -- 172.21.15.12:0/2440731819 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c105740 msgr2=0x7f3b4c107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.743+0000 7f3b52000700 1 --2- 172.21.15.12:0/2440731819 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c105740 0x7f3b4c107b30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3b38009990 tx=0x7f3b38009ca0 comp rx=0 tx=0).stop 2024-03-17T23:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.743+0000 7f3b52000700 1 -- 172.21.15.12:0/2440731819 shutdown_connections 2024-03-17T23:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.743+0000 7f3b52000700 1 --2- 172.21.15.12:0/2440731819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b4c108070 0x7f3b4c10a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.743+0000 7f3b52000700 1 --2- 172.21.15.12:0/2440731819 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b4c102e10 0x7f3b4c105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.743+0000 7f3b52000700 1 --2- 172.21.15.12:0/2440731819 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c105740 0x7f3b4c107b30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.743+0000 7f3b52000700 1 -- 172.21.15.12:0/2440731819 >> 172.21.15.12:0/2440731819 conn(0x7f3b4c0fc7c0 msgr2=0x7f3b4c0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.743+0000 7f3b52000700 1 -- 172.21.15.12:0/2440731819 shutdown_connections 2024-03-17T23:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.743+0000 7f3b52000700 1 -- 172.21.15.12:0/2440731819 wait complete. 2024-03-17T23:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.744+0000 7f3b52000700 1 Processor -- start 2024-03-17T23:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.745+0000 7f3b52000700 1 -- start start 2024-03-17T23:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.745+0000 7f3b52000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b4c102e10 0x7f3b4c19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.745+0000 7f3b52000700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c105740 0x7f3b4c19abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.745+0000 7f3b52000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b4c108070 0x7f3b4c1a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.745+0000 7f3b52000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4c10df30 con 0x7f3b4c108070 2024-03-17T23:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.745+0000 7f3b52000700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4c10ddb0 con 0x7f3b4c105740 2024-03-17T23:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.745+0000 7f3b52000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4c10e0b0 con 0x7f3b4c102e10 2024-03-17T23:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.745+0000 7f3b4aa00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c105740 0x7f3b4c19abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.745+0000 7f3b4b400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b4c102e10 0x7f3b4c19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.745+0000 7f3b4aa00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c105740 0x7f3b4c19abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:55698/0 (socket says 172.21.15.12:55698) 2024-03-17T23:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.745+0000 7f3b4aa00700 1 -- 172.21.15.12:0/2620678576 learned_addr learned my addr 172.21.15.12:0/2620678576 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.746+0000 7f3b4be00700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b4c108070 0x7f3b4c1a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.746+0000 7f3b4b400700 1 -- 172.21.15.12:0/2620678576 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c105740 msgr2=0x7f3b4c19abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.746+0000 7f3b4b400700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c105740 0x7f3b4c19abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.746+0000 7f3b4b400700 1 -- 172.21.15.12:0/2620678576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b4c108070 msgr2=0x7f3b4c1a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.746+0000 7f3b4b400700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b4c108070 0x7f3b4c1a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.746+0000 7f3b4b400700 1 -- 172.21.15.12:0/2620678576 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b4c1a22f0 con 0x7f3b4c102e10 2024-03-17T23:55:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.746+0000 7f3b4be00700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b4c108070 0x7f3b4c1a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:55:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.746+0000 7f3b4aa00700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c105740 0x7f3b4c19abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:55:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.747+0000 7f3b4b400700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b4c102e10 0x7f3b4c19a690 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00ea20 tx=0x7f3b3c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.747+0000 7f3b43e00700 1 -- 172.21.15.12:0/2620678576 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c00cbd0 con 0x7f3b4c102e10 2024-03-17T23:55:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.747+0000 7f3b43e00700 1 -- 172.21.15.12:0/2620678576 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b3c00cd70 con 0x7f3b4c102e10 2024-03-17T23:55:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.747+0000 7f3b52000700 1 -- 172.21.15.12:0/2620678576 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b4c1a25e0 con 0x7f3b4c102e10 2024-03-17T23:55:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.747+0000 7f3b43e00700 1 -- 172.21.15.12:0/2620678576 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c018660 con 0x7f3b4c102e10 2024-03-17T23:55:33.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.747+0000 7f3b52000700 1 -- 172.21.15.12:0/2620678576 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b4c1a2b70 con 0x7f3b4c102e10 2024-03-17T23:55:33.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.749+0000 7f3b52000700 1 -- 172.21.15.12:0/2620678576 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b2c005330 con 0x7f3b4c102e10 2024-03-17T23:55:33.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.750+0000 7f3b43e00700 1 -- 172.21.15.12:0/2620678576 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3b3c018800 con 0x7f3b4c102e10 2024-03-17T23:55:33.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.752+0000 7f3b43e00700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3b300a2a50 0x7f3b300a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:33.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.752+0000 7f3b43e00700 1 -- 172.21.15.12:0/2620678576 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f3b3c0cefa0 con 0x7f3b4c102e10 2024-03-17T23:55:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.752+0000 7f3b4aa00700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3b300a2a50 0x7f3b300a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.753+0000 7f3b4aa00700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3b300a2a50 0x7f3b300a4f10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3b38006010 tx=0x7f3b3800b8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:33.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:33.756+0000 7f3b43e00700 1 -- 172.21.15.12:0/2620678576 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3b3c09bc40 con 0x7f3b4c102e10 2024-03-17T23:55:34.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.083+0000 7f3b52000700 1 -- 172.21.15.12:0/2620678576 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f3b2c005620 con 0x7f3b4c102e10 2024-03-17T23:55:34.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.090+0000 7f3b43e00700 1 -- 172.21.15.12:0/2620678576 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f3b3c09b540 con 0x7f3b4c102e10 2024-03-17T23:55:34.092 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-03-17T23:55:34.092 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQD2gvdlXh0nBRAAeocix39bmwdiLBKlZMPZmw== 2024-03-17T23:55:34.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.094+0000 7f3b52000700 1 -- 172.21.15.12:0/2620678576 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3b300a2a50 msgr2=0x7f3b300a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:34.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.094+0000 7f3b52000700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3b300a2a50 0x7f3b300a4f10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3b38006010 tx=0x7f3b3800b8a0 comp rx=0 tx=0).stop 2024-03-17T23:55:34.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.094+0000 7f3b52000700 1 -- 172.21.15.12:0/2620678576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b4c102e10 msgr2=0x7f3b4c19a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:34.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.094+0000 7f3b52000700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b4c102e10 0x7f3b4c19a690 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00ea20 tx=0x7f3b3c00ed30 comp rx=0 tx=0).stop 2024-03-17T23:55:34.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.094+0000 7f3b52000700 1 -- 172.21.15.12:0/2620678576 shutdown_connections 2024-03-17T23:55:34.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.094+0000 7f3b52000700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3b300a2a50 0x7f3b300a4f10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:34.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.094+0000 7f3b52000700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b4c108070 0x7f3b4c1a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:34.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.094+0000 7f3b52000700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b4c102e10 0x7f3b4c19a690 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:34.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.094+0000 7f3b52000700 1 --2- 172.21.15.12:0/2620678576 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c105740 0x7f3b4c19abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:34.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.094+0000 7f3b52000700 1 -- 172.21.15.12:0/2620678576 >> 172.21.15.12:0/2620678576 conn(0x7f3b4c0fc7c0 msgr2=0x7f3b4c102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:34.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.094+0000 7f3b52000700 1 -- 172.21.15.12:0/2620678576 shutdown_connections 2024-03-17T23:55:34.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:34.094+0000 7f3b52000700 1 -- 172.21.15.12:0/2620678576 wait complete. 2024-03-17T23:55:34.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:34 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2620678576' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-17T23:55:34.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:34 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-17T23:55:34.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:34 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-17T23:55:34.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:34 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2620678576' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-17T23:55:34.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:34 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-17T23:55:34.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:34 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-17T23:55:34.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:34 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2620678576' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-17T23:55:34.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:34 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-17T23:55:34.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:34 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-17T23:55:34.954 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:55:34.954 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-17T23:55:34.954 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-17T23:55:35.123 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-17T23:55:35.123 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-17T23:55:35.123 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-17T23:55:35.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:35 smithi079 ceph-mon[36241]: pgmap v340: 129 pgs: 129 active+clean; 629 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.4 KiB/s wr, 20 op/s 2024-03-17T23:55:35.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:35 smithi191 ceph-mon[36708]: pgmap v340: 129 pgs: 129 active+clean; 629 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.4 KiB/s wr, 20 op/s 2024-03-17T23:55:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:35.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:35 smithi012 ceph-mon[31541]: pgmap v340: 129 pgs: 129 active+clean; 629 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.4 KiB/s wr, 20 op/s 2024-03-17T23:55:37.467 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:37 smithi012 ceph-mon[31541]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 9.7 KiB/s wr, 24 op/s 2024-03-17T23:55:37.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:37 smithi079 ceph-mon[36241]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 9.7 KiB/s wr, 24 op/s 2024-03-17T23:55:37.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:37 smithi191 ceph-mon[36708]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 9.7 KiB/s wr, 24 op/s 2024-03-17T23:55:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.781+0000 7ff269800700 1 -- 172.21.15.12:0/480797650 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff264103ca0 msgr2=0x7ff26410c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.781+0000 7ff269800700 1 --2- 172.21.15.12:0/480797650 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff264103ca0 0x7ff26410c190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff250009990 tx=0x7ff250009ca0 comp rx=0 tx=0).stop 2024-03-17T23:55:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.781+0000 7ff269800700 1 -- 172.21.15.12:0/480797650 shutdown_connections 2024-03-17T23:55:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.781+0000 7ff269800700 1 --2- 172.21.15.12:0/480797650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff26410c7b0 0x7ff26410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.781+0000 7ff269800700 1 --2- 172.21.15.12:0/480797650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff264103380 0x7ff264103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:37.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.781+0000 7ff269800700 1 --2- 172.21.15.12:0/480797650 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff264103ca0 0x7ff26410c190 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:37.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.781+0000 7ff269800700 1 -- 172.21.15.12:0/480797650 >> 172.21.15.12:0/480797650 conn(0x7ff2640fcbb0 msgr2=0x7ff2640fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:37.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.781+0000 7ff269800700 1 -- 172.21.15.12:0/480797650 shutdown_connections 2024-03-17T23:55:37.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.781+0000 7ff269800700 1 -- 172.21.15.12:0/480797650 wait complete. 2024-03-17T23:55:37.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.782+0000 7ff269800700 1 Processor -- start 2024-03-17T23:55:37.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.783+0000 7ff269800700 1 -- start start 2024-03-17T23:55:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.783+0000 7ff269800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff264103380 0x7ff26419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.783+0000 7ff269800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff264103ca0 0x7ff26419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.783+0000 7ff269800700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff26410c7b0 0x7ff264198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.783+0000 7ff269800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff2641124d0 con 0x7ff264103380 2024-03-17T23:55:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.783+0000 7ff269800700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff264112350 con 0x7ff26410c7b0 2024-03-17T23:55:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.783+0000 7ff269800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff264112650 con 0x7ff264103ca0 2024-03-17T23:55:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.784+0000 7ff263e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff264103380 0x7ff26419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.784+0000 7ff263400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff264103ca0 0x7ff26419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.784+0000 7ff268e00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff26410c7b0 0x7ff264198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.784+0000 7ff263400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff264103ca0 0x7ff26419ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:55038/0 (socket says 172.21.15.12:55038) 2024-03-17T23:55:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.784+0000 7ff263400700 1 -- 172.21.15.12:0/4090494609 learned_addr learned my addr 172.21.15.12:0/4090494609 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.784+0000 7ff263400700 1 -- 172.21.15.12:0/4090494609 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff26410c7b0 msgr2=0x7ff264198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.784+0000 7ff263400700 1 --2- 172.21.15.12:0/4090494609 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff26410c7b0 0x7ff264198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.784+0000 7ff263400700 1 -- 172.21.15.12:0/4090494609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff264103380 msgr2=0x7ff26419e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.784+0000 7ff263400700 1 --2- 172.21.15.12:0/4090494609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff264103380 0x7ff26419e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.784+0000 7ff263400700 1 -- 172.21.15.12:0/4090494609 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff264199030 con 0x7ff264103ca0 2024-03-17T23:55:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.784+0000 7ff268e00700 1 --2- 172.21.15.12:0/4090494609 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff26410c7b0 0x7ff264198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:55:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.784+0000 7ff263400700 1 --2- 172.21.15.12:0/4090494609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff264103ca0 0x7ff26419ebc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff250000c00 tx=0x7ff25000bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.785+0000 7ff260c00700 1 -- 172.21.15.12:0/4090494609 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff250027070 con 0x7ff264103ca0 2024-03-17T23:55:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.785+0000 7ff260c00700 1 -- 172.21.15.12:0/4090494609 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff250003d70 con 0x7ff264103ca0 2024-03-17T23:55:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.785+0000 7ff260c00700 1 -- 172.21.15.12:0/4090494609 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25000f460 con 0x7ff264103ca0 2024-03-17T23:55:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.785+0000 7ff269800700 1 -- 172.21.15.12:0/4090494609 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2641992c0 con 0x7ff264103ca0 2024-03-17T23:55:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.785+0000 7ff269800700 1 -- 172.21.15.12:0/4090494609 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2641a54e0 con 0x7ff264103ca0 2024-03-17T23:55:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.786+0000 7ff269800700 1 -- 172.21.15.12:0/4090494609 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff26406b0f0 con 0x7ff264103ca0 2024-03-17T23:55:37.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.788+0000 7ff260c00700 1 -- 172.21.15.12:0/4090494609 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff25002a070 con 0x7ff264103ca0 2024-03-17T23:55:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.790+0000 7ff260c00700 1 --2- 172.21.15.12:0/4090494609 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff2480a29a0 0x7ff2480a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.790+0000 7ff260c00700 1 -- 172.21.15.12:0/4090494609 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff2500d0480 con 0x7ff264103ca0 2024-03-17T23:55:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.790+0000 7ff263e00700 1 --2- 172.21.15.12:0/4090494609 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff2480a29a0 0x7ff2480a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.791+0000 7ff263e00700 1 --2- 172.21.15.12:0/4090494609 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff2480a29a0 0x7ff2480a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff254009c60 tx=0x7ff254009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:37.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:37.792+0000 7ff260c00700 1 -- 172.21.15.12:0/4090494609 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff250031070 con 0x7ff264103ca0 2024-03-17T23:55:38.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.121+0000 7ff269800700 1 -- 172.21.15.12:0/4090494609 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ff26402db70 con 0x7ff264103ca0 2024-03-17T23:55:38.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.130+0000 7ff260c00700 1 -- 172.21.15.12:0/4090494609 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262661 (secure 0 0 0) 0x7ff250013070 con 0x7ff264103ca0 2024-03-17T23:55:38.136 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.136+0000 7ff269800700 1 -- 172.21.15.12:0/4090494609 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff2480a29a0 msgr2=0x7ff2480a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.136+0000 7ff269800700 1 --2- 172.21.15.12:0/4090494609 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff2480a29a0 0x7ff2480a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff254009c60 tx=0x7ff254009260 comp rx=0 tx=0).stop 2024-03-17T23:55:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.136+0000 7ff269800700 1 -- 172.21.15.12:0/4090494609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff264103ca0 msgr2=0x7ff26419ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.136+0000 7ff269800700 1 --2- 172.21.15.12:0/4090494609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff264103ca0 0x7ff26419ebc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff250000c00 tx=0x7ff25000bab0 comp rx=0 tx=0).stop 2024-03-17T23:55:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.137+0000 7ff269800700 1 -- 172.21.15.12:0/4090494609 shutdown_connections 2024-03-17T23:55:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.137+0000 7ff269800700 1 --2- 172.21.15.12:0/4090494609 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7ff2480a29a0 0x7ff2480a4e60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.137+0000 7ff269800700 1 --2- 172.21.15.12:0/4090494609 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff26410c7b0 0x7ff264198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.137+0000 7ff269800700 1 --2- 172.21.15.12:0/4090494609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff264103380 0x7ff26419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.137+0000 7ff269800700 1 --2- 172.21.15.12:0/4090494609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff264103ca0 0x7ff26419ebc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.137+0000 7ff269800700 1 -- 172.21.15.12:0/4090494609 >> 172.21.15.12:0/4090494609 conn(0x7ff2640fcbb0 msgr2=0x7ff2640fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.137+0000 7ff269800700 1 -- 172.21.15.12:0/4090494609 shutdown_connections 2024-03-17T23:55:38.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:38.137+0000 7ff269800700 1 -- 172.21.15.12:0/4090494609 wait complete. 2024-03-17T23:55:38.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:38 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4090494609' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-17T23:55:38.492 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:38 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/4090494609' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-17T23:55:38.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:38 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/4090494609' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-17T23:55:38.959 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3179836366},{"type":"v1","addr":"172.21.15.12:6801","nonce":3179836366}]},"active_addr":"172.21.15.12:6801/3179836366","active_change":"2024-03-17T23:46:53.956106+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.12:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":1070514851}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":1849534255}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":3450532080}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":1081321261}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":4193772272}]}]} 2024-03-17T23:55:38.964 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-17T23:55:38.964 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-17T23:55:38.964 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-17T23:55:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:39.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:39 smithi079 ceph-mon[36241]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.1 KiB/s wr, 22 op/s 2024-03-17T23:55:39.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:39 smithi191 ceph-mon[36708]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.1 KiB/s wr, 22 op/s 2024-03-17T23:55:39.632 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:39 smithi012 ceph-mon[31541]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.1 KiB/s wr, 22 op/s 2024-03-17T23:55:41.416 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:41 smithi012 ceph-mon[31541]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 21 op/s 2024-03-17T23:55:41.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:41 smithi079 ceph-mon[36241]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 21 op/s 2024-03-17T23:55:41.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:41 smithi191 ceph-mon[36708]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 21 op/s 2024-03-17T23:55:41.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.720+0000 7f3db3800700 1 -- 172.21.15.12:0/3060727956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dac107650 msgr2=0x7f3dac10f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:41.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.720+0000 7f3db3800700 1 --2- 172.21.15.12:0/3060727956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dac107650 0x7f3dac10f220 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3da4009fc0 tx=0x7f3da400b330 comp rx=0 tx=0).stop 2024-03-17T23:55:41.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.720+0000 7f3db3800700 1 -- 172.21.15.12:0/3060727956 shutdown_connections 2024-03-17T23:55:41.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.720+0000 7f3db3800700 1 --2- 172.21.15.12:0/3060727956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dac107650 0x7f3dac10f220 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:41.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.720+0000 7f3db3800700 1 --2- 172.21.15.12:0/3060727956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dac102580 0x7f3dac102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:41.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.720+0000 7f3db3800700 1 --2- 172.21.15.12:0/3060727956 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dac102f30 0x7f3dac106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:41.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.720+0000 7f3db3800700 1 -- 172.21.15.12:0/3060727956 >> 172.21.15.12:0/3060727956 conn(0x7f3dac079430 msgr2=0x7f3dac079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:41.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.721+0000 7f3db3800700 1 -- 172.21.15.12:0/3060727956 shutdown_connections 2024-03-17T23:55:41.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.721+0000 7f3db3800700 1 -- 172.21.15.12:0/3060727956 wait complete. 2024-03-17T23:55:41.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.722+0000 7f3db3800700 1 Processor -- start 2024-03-17T23:55:41.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.722+0000 7f3db3800700 1 -- start start 2024-03-17T23:55:41.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.722+0000 7f3db3800700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dac102580 0x7f3dac0745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:41.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.722+0000 7f3db3800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dac102f30 0x7f3dac074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:41.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.722+0000 7f3db3800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dac107650 0x7f3dac075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:41.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.722+0000 7f3db3800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac1126a0 con 0x7f3dac102f30 2024-03-17T23:55:41.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.722+0000 7f3db3800700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac112520 con 0x7f3dac102580 2024-03-17T23:55:41.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.722+0000 7f3db3800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac112820 con 0x7f3dac107650 2024-03-17T23:55:41.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.723+0000 7f3db0e00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dac102580 0x7f3dac0745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:41.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.723+0000 7f3db0e00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dac102580 0x7f3dac0745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:49044/0 (socket says 172.21.15.12:49044) 2024-03-17T23:55:41.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.723+0000 7f3db0e00700 1 -- 172.21.15.12:0/167309418 learned_addr learned my addr 172.21.15.12:0/167309418 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:41.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.723+0000 7f3dabe00700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dac102f30 0x7f3dac074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:41.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.723+0000 7f3db1800700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dac107650 0x7f3dac075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:41.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.723+0000 7f3db0e00700 1 -- 172.21.15.12:0/167309418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dac107650 msgr2=0x7f3dac075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:41.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.723+0000 7f3db0e00700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dac107650 0x7f3dac075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:41.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.723+0000 7f3db0e00700 1 -- 172.21.15.12:0/167309418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dac102f30 msgr2=0x7f3dac074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:41.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.723+0000 7f3db0e00700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dac102f30 0x7f3dac074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:41.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.723+0000 7f3db0e00700 1 -- 172.21.15.12:0/167309418 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac1a7060 con 0x7f3dac102580 2024-03-17T23:55:41.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.723+0000 7f3db1800700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dac107650 0x7f3dac075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:55:41.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.724+0000 7f3db0e00700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dac102580 0x7f3dac0745d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3da000e9a0 tx=0x7f3da000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:41.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.724+0000 7f3da9600700 1 -- 172.21.15.12:0/167309418 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da000cbd0 con 0x7f3dac102580 2024-03-17T23:55:41.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.724+0000 7f3db3800700 1 -- 172.21.15.12:0/167309418 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dac1a7350 con 0x7f3dac102580 2024-03-17T23:55:41.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.724+0000 7f3da9600700 1 -- 172.21.15.12:0/167309418 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da000cd70 con 0x7f3dac102580 2024-03-17T23:55:41.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.724+0000 7f3da9600700 1 -- 172.21.15.12:0/167309418 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0018710 con 0x7f3dac102580 2024-03-17T23:55:41.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.724+0000 7f3db3800700 1 -- 172.21.15.12:0/167309418 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dac1a7890 con 0x7f3dac102580 2024-03-17T23:55:41.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.725+0000 7f3db3800700 1 -- 172.21.15.12:0/167309418 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dac068f90 con 0x7f3dac102580 2024-03-17T23:55:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.727+0000 7f3da9600700 1 -- 172.21.15.12:0/167309418 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3da0010430 con 0x7f3dac102580 2024-03-17T23:55:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.728+0000 7f3da9600700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3d940a2920 0x7f3d940a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.729+0000 7f3da9600700 1 -- 172.21.15.12:0/167309418 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f3da00cf170 con 0x7f3dac102580 2024-03-17T23:55:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.729+0000 7f3dabe00700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3d940a2920 0x7f3d940a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.730+0000 7f3dabe00700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3d940a2920 0x7f3d940a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3dac108a90 tx=0x7f3d98009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.737+0000 7f3da9600700 1 -- 172.21.15.12:0/167309418 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3da009be10 con 0x7f3dac102580 2024-03-17T23:55:41.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.971+0000 7f3db3800700 1 -- 172.21.15.12:0/167309418 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3dac02db70 con 0x7f3dac102580 2024-03-17T23:55:41.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.972+0000 7f3da9600700 1 -- 172.21.15.12:0/167309418 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24044 (secure 0 0 0) 0x7f3da009b710 con 0x7f3dac102580 2024-03-17T23:55:41.974 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:41.975 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":90,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:55:02.337441+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:55:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.977+0000 7f3db3800700 1 -- 172.21.15.12:0/167309418 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3d940a2920 msgr2=0x7f3d940a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:41.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.977+0000 7f3db3800700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3d940a2920 0x7f3d940a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3dac108a90 tx=0x7f3d98009310 comp rx=0 tx=0).stop 2024-03-17T23:55:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.977+0000 7f3db3800700 1 -- 172.21.15.12:0/167309418 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dac102580 msgr2=0x7f3dac0745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.977+0000 7f3db3800700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dac102580 0x7f3dac0745d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3da000e9a0 tx=0x7f3da000ed60 comp rx=0 tx=0).stop 2024-03-17T23:55:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.977+0000 7f3db3800700 1 -- 172.21.15.12:0/167309418 shutdown_connections 2024-03-17T23:55:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.977+0000 7f3db3800700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3d940a2920 0x7f3d940a4de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.977+0000 7f3db3800700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dac107650 0x7f3dac075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.977+0000 7f3db3800700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dac102580 0x7f3dac0745d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.977+0000 7f3db3800700 1 --2- 172.21.15.12:0/167309418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3dac102f30 0x7f3dac074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:41.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.977+0000 7f3db3800700 1 -- 172.21.15.12:0/167309418 >> 172.21.15.12:0/167309418 conn(0x7f3dac079430 msgr2=0x7f3dac1002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:41.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.978+0000 7f3db3800700 1 -- 172.21.15.12:0/167309418 shutdown_connections 2024-03-17T23:55:41.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:41.978+0000 7f3db3800700 1 -- 172.21.15.12:0/167309418 wait complete. 2024-03-17T23:55:42.369 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:42 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/167309418' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:42.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:42 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/167309418' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:42.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:42 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/167309418' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:42.795 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-17T23:55:42.796 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-17T23:55:43.292 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:43.465 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:43 smithi012 ceph-mon[31541]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 KiB/s wr, 13 op/s 2024-03-17T23:55:43.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:43 smithi079 ceph-mon[36241]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 KiB/s wr, 13 op/s 2024-03-17T23:55:43.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:43 smithi191 ceph-mon[36708]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 KiB/s wr, 13 op/s 2024-03-17T23:55:45.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.506+0000 7f94c2c00700 1 -- 172.21.15.12:0/1079791910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94bc06a920 msgr2=0x7f94bc10f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:45.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.506+0000 7f94c2c00700 1 --2- 172.21.15.12:0/1079791910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94bc06a920 0x7f94bc10f160 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f94b400b0a0 tx=0x7f94b400b3b0 comp rx=0 tx=0).stop 2024-03-17T23:55:45.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.506+0000 7f94c2c00700 1 -- 172.21.15.12:0/1079791910 shutdown_connections 2024-03-17T23:55:45.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.506+0000 7f94c2c00700 1 --2- 172.21.15.12:0/1079791910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94bc06a920 0x7f94bc10f160 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:45.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.506+0000 7f94c2c00700 1 --2- 172.21.15.12:0/1079791910 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94bc108320 0x7f94bc108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:45.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.506+0000 7f94c2c00700 1 --2- 172.21.15.12:0/1079791910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94bc069f80 0x7f94bc06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:45.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.506+0000 7f94c2c00700 1 -- 172.21.15.12:0/1079791910 >> 172.21.15.12:0/1079791910 conn(0x7f94bc078100 msgr2=0x7f94bc078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:45.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.506+0000 7f94c2c00700 1 -- 172.21.15.12:0/1079791910 shutdown_connections 2024-03-17T23:55:45.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.506+0000 7f94c2c00700 1 -- 172.21.15.12:0/1079791910 wait complete. 2024-03-17T23:55:45.510 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:45 smithi012 ceph-mon[31541]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-17T23:55:45.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.507+0000 7f94c2c00700 1 Processor -- start 2024-03-17T23:55:45.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.507+0000 7f94c2c00700 1 -- start start 2024-03-17T23:55:45.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94c2c00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94bc069f80 0x7f94bc1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:45.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94c2c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94bc06a920 0x7f94bc1a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:45.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94c2c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94bc108320 0x7f94bc19aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:45.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94c2c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc1125c0 con 0x7f94bc06a920 2024-03-17T23:55:45.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94c2c00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc112440 con 0x7f94bc069f80 2024-03-17T23:55:45.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94c2c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc112740 con 0x7f94bc108320 2024-03-17T23:55:45.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94bbe00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94bc069f80 0x7f94bc1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:45.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94c0c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94bc108320 0x7f94bc19aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:45.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94bbe00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94bc069f80 0x7f94bc1a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:47606/0 (socket says 172.21.15.12:47606) 2024-03-17T23:55:45.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94c0c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94bc108320 0x7f94bc19aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:59710/0 (socket says 172.21.15.12:59710) 2024-03-17T23:55:45.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94bbe00700 1 -- 172.21.15.12:0/1128520674 learned_addr learned my addr 172.21.15.12:0/1128520674 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:45.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94bb400700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94bc06a920 0x7f94bc1a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:45.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94bbe00700 1 -- 172.21.15.12:0/1128520674 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94bc108320 msgr2=0x7f94bc19aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:45.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94bbe00700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94bc108320 0x7f94bc19aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94bbe00700 1 -- 172.21.15.12:0/1128520674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94bc06a920 msgr2=0x7f94bc1a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94bbe00700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94bc06a920 0x7f94bc1a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94bbe00700 1 -- 172.21.15.12:0/1128520674 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc19b1b0 con 0x7f94bc069f80 2024-03-17T23:55:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.508+0000 7f94bb400700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94bc06a920 0x7f94bc1a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:55:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.509+0000 7f94bbe00700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94bc069f80 0x7f94bc1a0890 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00e9a0 tx=0x7f94ac00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.509+0000 7f94b8c00700 1 -- 172.21.15.12:0/1128520674 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac00cbd0 con 0x7f94bc069f80 2024-03-17T23:55:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.509+0000 7f94b8c00700 1 -- 172.21.15.12:0/1128520674 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94ac00cd70 con 0x7f94bc069f80 2024-03-17T23:55:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.509+0000 7f94b8c00700 1 -- 172.21.15.12:0/1128520674 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac018750 con 0x7f94bc069f80 2024-03-17T23:55:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.509+0000 7f94c2c00700 1 -- 172.21.15.12:0/1128520674 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94bc19b4a0 con 0x7f94bc069f80 2024-03-17T23:55:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.509+0000 7f94c2c00700 1 -- 172.21.15.12:0/1128520674 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94bc1a7760 con 0x7f94bc069f80 2024-03-17T23:55:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.510+0000 7f94c2c00700 1 -- 172.21.15.12:0/1128520674 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94bc068f90 con 0x7f94bc069f80 2024-03-17T23:55:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.513+0000 7f94b8c00700 1 -- 172.21.15.12:0/1128520674 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f94ac010430 con 0x7f94bc069f80 2024-03-17T23:55:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.514+0000 7f94b8c00700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f94a00a29a0 0x7f94a00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.515+0000 7f94b8c00700 1 -- 172.21.15.12:0/1128520674 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f94ac0cf070 con 0x7f94bc069f80 2024-03-17T23:55:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.515+0000 7f94bb400700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f94a00a29a0 0x7f94a00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.515+0000 7f94bb400700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f94a00a29a0 0x7f94a00a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f94bc1031d0 tx=0x7f94a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:45.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.516+0000 7f94b8c00700 1 -- 172.21.15.12:0/1128520674 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f94ac09bd10 con 0x7f94bc069f80 2024-03-17T23:55:45.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:45 smithi079 ceph-mon[36241]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-17T23:55:45.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:45 smithi191 ceph-mon[36708]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-17T23:55:45.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.757+0000 7f94c2c00700 1 -- 172.21.15.12:0/1128520674 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f94bc1a79e0 con 0x7f94bc069f80 2024-03-17T23:55:45.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.760+0000 7f94b8c00700 1 -- 172.21.15.12:0/1128520674 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24044 (secure 0 0 0) 0x7f94ac09b610 con 0x7f94bc069f80 2024-03-17T23:55:45.762 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:55:45.762 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":90,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:55:02.337441+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:55:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.764+0000 7f94c2c00700 1 -- 172.21.15.12:0/1128520674 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f94a00a29a0 msgr2=0x7f94a00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.764+0000 7f94c2c00700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f94a00a29a0 0x7f94a00a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f94bc1031d0 tx=0x7f94a8009260 comp rx=0 tx=0).stop 2024-03-17T23:55:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.764+0000 7f94c2c00700 1 -- 172.21.15.12:0/1128520674 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94bc069f80 msgr2=0x7f94bc1a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.764+0000 7f94c2c00700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94bc069f80 0x7f94bc1a0890 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00e9a0 tx=0x7f94ac00ecb0 comp rx=0 tx=0).stop 2024-03-17T23:55:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.764+0000 7f94c2c00700 1 -- 172.21.15.12:0/1128520674 shutdown_connections 2024-03-17T23:55:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.764+0000 7f94c2c00700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f94a00a29a0 0x7f94a00a4e60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.764+0000 7f94c2c00700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94bc108320 0x7f94bc19aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.764+0000 7f94c2c00700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94bc069f80 0x7f94bc1a0890 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.764+0000 7f94c2c00700 1 --2- 172.21.15.12:0/1128520674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94bc06a920 0x7f94bc1a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.764+0000 7f94c2c00700 1 -- 172.21.15.12:0/1128520674 >> 172.21.15.12:0/1128520674 conn(0x7f94bc078100 msgr2=0x7f94bc10d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.765+0000 7f94c2c00700 1 -- 172.21.15.12:0/1128520674 shutdown_connections 2024-03-17T23:55:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:45.765+0000 7f94c2c00700 1 -- 172.21.15.12:0/1128520674 wait complete. 2024-03-17T23:55:46.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:46 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1128520674' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:46.562 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:46 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1128520674' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:46.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:46 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1128520674' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:55:46.646 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-17T23:55:46.647 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-17T23:55:46.647 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-17T23:55:46.648 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-17T23:55:46.648 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-17T23:55:46.648 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-17T23:55:46.649 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-17T23:55:46.649 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-17T23:55:46.650 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-17T23:55:46.650 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-17T23:55:46.650 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-17T23:55:46.651 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-17T23:55:47.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:47 smithi079 ceph-mon[36241]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-17T23:55:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:47 smithi191 ceph-mon[36708]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-17T23:55:47.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:47 smithi012 ceph-mon[31541]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-17T23:55:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:48.900 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:48.902 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:48.928 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:48.935 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:48.935 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:48.936 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:48.936 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:48.936 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:48.951 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:55:49.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:49 smithi079 ceph-mon[36241]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-17T23:55:49.796 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:49 smithi012 ceph-mon[31541]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-17T23:55:49.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:49 smithi191 ceph-mon[36708]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-17T23:55:51.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:51 smithi079 ceph-mon[36241]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-17T23:55:51.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:51 smithi191 ceph-mon[36708]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-17T23:55:51.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:51 smithi012 ceph-mon[31541]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-17T23:55:53.725 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:53 smithi012 ceph-mon[31541]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-17T23:55:53.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:53 smithi079 ceph-mon[36241]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-17T23:55:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:53 smithi191 ceph-mon[36708]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-17T23:55:54.730 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-17T23:55:54.730 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-17T23:55:54.730 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-17T23:55:54.730 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-03-17T23:55:54.730 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-03-17T23:55:54.730 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-17T23:55:54.730 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-03-17T23:55:54.731 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-03-17T23:55:54.731 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-17T23:55:54.731 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-03-17T23:55:54.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-17T23:55:54.797 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-17T23:55:54.798 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-17T23:55:54.798 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-03-17T23:55:54.798 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-03-17T23:55:54.798 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-17T23:55:54.798 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-03-17T23:55:54.799 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-03-17T23:55:54.799 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-17T23:55:54.799 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-03-17T23:55:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-17T23:55:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-17T23:55:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-17T23:55:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-03-17T23:55:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-03-17T23:55:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-17T23:55:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-03-17T23:55:54.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-03-17T23:55:54.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-17T23:55:54.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-03-17T23:55:55.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:55 smithi012 ceph-mon[31541]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-17T23:55:55.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-17T23:55:55.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-17T23:55:55.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-17T23:55:55.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-03-17T23:55:55.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-03-17T23:55:55.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-17T23:55:55.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-03-17T23:55:55.745 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-03-17T23:55:55.745 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-17T23:55:55.745 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-03-17T23:55:55.745 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:55 smithi012 ceph-mon[31541]: osdmap e91: 12 total, 12 up, 12 in 2024-03-17T23:55:55.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:55 smithi079 ceph-mon[36241]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-17T23:55:55.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-17T23:55:55.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-17T23:55:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-17T23:55:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-03-17T23:55:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-03-17T23:55:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-17T23:55:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-03-17T23:55:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-03-17T23:55:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-17T23:55:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-03-17T23:55:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:55 smithi079 ceph-mon[36241]: osdmap e91: 12 total, 12 up, 12 in 2024-03-17T23:55:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:55 smithi191 ceph-mon[36708]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-17T23:55:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-17T23:55:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-17T23:55:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-17T23:55:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-03-17T23:55:55.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-03-17T23:55:55.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-17T23:55:55.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-03-17T23:55:55.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-03-17T23:55:55.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-17T23:55:55.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-03-17T23:55:55.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:55 smithi191 ceph-mon[36708]: osdmap e91: 12 total, 12 up, 12 in 2024-03-17T23:55:56.758 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:56 smithi012 ceph-mon[31541]: osdmap e92: 12 total, 12 up, 12 in 2024-03-17T23:55:56.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:56 smithi079 ceph-mon[36241]: osdmap e92: 12 total, 12 up, 12 in 2024-03-17T23:55:56.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:56 smithi191 ceph-mon[36708]: osdmap e92: 12 total, 12 up, 12 in 2024-03-17T23:55:57.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:57 smithi012 ceph-mon[31541]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-17T23:55:57.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.727+0000 7f872e200700 1 -- 172.21.15.12:0/3580385906 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8728069f80 msgr2=0x7f872806a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:57.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.727+0000 7f872e200700 1 --2- 172.21.15.12:0/3580385906 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8728069f80 0x7f872806a400 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f87140099c0 tx=0x7f8714009cd0 comp rx=0 tx=0).stop 2024-03-17T23:55:57.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.727+0000 7f872e200700 1 -- 172.21.15.12:0/3580385906 shutdown_connections 2024-03-17T23:55:57.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.727+0000 7f872e200700 1 --2- 172.21.15.12:0/3580385906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f872806a940 0x7f872810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:57.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.727+0000 7f872e200700 1 --2- 172.21.15.12:0/3580385906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8728108300 0x7f87281086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:57.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.727+0000 7f872e200700 1 --2- 172.21.15.12:0/3580385906 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8728069f80 0x7f872806a400 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:57.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.727+0000 7f872e200700 1 -- 172.21.15.12:0/3580385906 >> 172.21.15.12:0/3580385906 conn(0x7f8728077f80 msgr2=0x7f8728078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:57.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.727+0000 7f872e200700 1 -- 172.21.15.12:0/3580385906 shutdown_connections 2024-03-17T23:55:57.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.727+0000 7f872e200700 1 -- 172.21.15.12:0/3580385906 wait complete. 2024-03-17T23:55:57.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.728+0000 7f872e200700 1 Processor -- start 2024-03-17T23:55:57.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.728+0000 7f872e200700 1 -- start start 2024-03-17T23:55:57.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.729+0000 7f872e200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8728069f80 0x7f87281015f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:57.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.729+0000 7f872e200700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f872806a940 0x7f8728101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:57.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.729+0000 7f872e200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8728108300 0x7f8728102070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:57.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.729+0000 7f872e200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f87281125d0 con 0x7f8728108300 2024-03-17T23:55:57.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.729+0000 7f872e200700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8728112450 con 0x7f872806a940 2024-03-17T23:55:57.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.729+0000 7f872e200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8728112750 con 0x7f8728069f80 2024-03-17T23:55:57.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.729+0000 7f8727400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8728069f80 0x7f87281015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:57.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.729+0000 7f8726a00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f872806a940 0x7f8728101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:57.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.729+0000 7f8727400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8728069f80 0x7f87281015f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:33820/0 (socket says 172.21.15.12:33820) 2024-03-17T23:55:57.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.729+0000 7f8726a00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f872806a940 0x7f8728101b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36196/0 (socket says 172.21.15.12:36196) 2024-03-17T23:55:57.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.729+0000 7f8727e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8728108300 0x7f8728102070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:57.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.729+0000 7f8726a00700 1 -- 172.21.15.12:0/135859831 learned_addr learned my addr 172.21.15.12:0/135859831 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:55:57.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.730+0000 7f8727400700 1 -- 172.21.15.12:0/135859831 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f872806a940 msgr2=0x7f8728101b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:57.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.730+0000 7f8727400700 1 --2- 172.21.15.12:0/135859831 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f872806a940 0x7f8728101b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.730+0000 7f8727400700 1 -- 172.21.15.12:0/135859831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8728108300 msgr2=0x7f8728102070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.730+0000 7f8727400700 1 --2- 172.21.15.12:0/135859831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8728108300 0x7f8728102070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.730+0000 7f8727400700 1 -- 172.21.15.12:0/135859831 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87281a7070 con 0x7f8728069f80 2024-03-17T23:55:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.731+0000 7f8727400700 1 --2- 172.21.15.12:0/135859831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8728069f80 0x7f87281015f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f871800e9f0 tx=0x7f871800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.731+0000 7f871fe00700 1 -- 172.21.15.12:0/135859831 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871800cbd0 con 0x7f8728069f80 2024-03-17T23:55:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.731+0000 7f871fe00700 1 -- 172.21.15.12:0/135859831 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f871800cd70 con 0x7f8728069f80 2024-03-17T23:55:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.731+0000 7f871fe00700 1 -- 172.21.15.12:0/135859831 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718018710 con 0x7f8728069f80 2024-03-17T23:55:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.731+0000 7f872e200700 1 -- 172.21.15.12:0/135859831 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87281a7360 con 0x7f8728069f80 2024-03-17T23:55:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.731+0000 7f872e200700 1 -- 172.21.15.12:0/135859831 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87281a7850 con 0x7f8728069f80 2024-03-17T23:55:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.732+0000 7f872e200700 1 -- 172.21.15.12:0/135859831 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f87280629e0 con 0x7f8728069f80 2024-03-17T23:55:57.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.736+0000 7f871fe00700 1 -- 172.21.15.12:0/135859831 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8718010430 con 0x7f8728069f80 2024-03-17T23:55:57.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.737+0000 7f871fe00700 1 --2- 172.21.15.12:0/135859831 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f870c0a29a0 0x7f870c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.738+0000 7f871fe00700 1 -- 172.21.15.12:0/135859831 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f87180cf220 con 0x7f8728069f80 2024-03-17T23:55:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.738+0000 7f871fe00700 1 --2- 172.21.15.12:0/135859831 >> [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359] conn(0x7f870c0adb00 0x7f870c0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:55:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.738+0000 7f871fe00700 1 -- 172.21.15.12:0/135859831 --> [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8718015e80 con 0x7f870c0adb00 2024-03-17T23:55:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.738+0000 7f871fe00700 1 -- 172.21.15.12:0/135859831 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f87180cf5d0 con 0x7f8728069f80 2024-03-17T23:55:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.738+0000 7f8726a00700 1 --2- 172.21.15.12:0/135859831 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f870c0a29a0 0x7f870c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.739+0000 7f8727e00700 1 --2- 172.21.15.12:0/135859831 >> [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359] conn(0x7f870c0adb00 0x7f870c0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:55:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.740+0000 7f8726a00700 1 --2- 172.21.15.12:0/135859831 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f870c0a29a0 0x7f870c0a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8728107180 tx=0x7f871401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.740+0000 7f8727e00700 1 --2- 172.21.15.12:0/135859831 >> [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359] conn(0x7f870c0adb00 0x7f870c0affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:55:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.741+0000 7f871fe00700 1 -- 172.21.15.12:0/135859831 <== osd.10 v2:172.21.15.79:6824/3620799359 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8718015e80 con 0x7f870c0adb00 2024-03-17T23:55:57.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.777+0000 7f872e200700 1 -- 172.21.15.12:0/135859831 --> [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8728062e30 con 0x7f870c0adb00 2024-03-17T23:55:57.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.778+0000 7f871fe00700 1 -- 172.21.15.12:0/135859831 <== osd.10 v2:172.21.15.79:6824/3620799359 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8728062e30 con 0x7f870c0adb00 2024-03-17T23:55:57.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.778+0000 7f872e200700 1 -- 172.21.15.12:0/135859831 >> [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359] conn(0x7f870c0adb00 msgr2=0x7f870c0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:57.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.778+0000 7f872e200700 1 --2- 172.21.15.12:0/135859831 >> [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359] conn(0x7f870c0adb00 0x7f870c0affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:55:57.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.779+0000 7f872e200700 1 -- 172.21.15.12:0/135859831 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f870c0a29a0 msgr2=0x7f870c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:57.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.779+0000 7f872e200700 1 --2- 172.21.15.12:0/135859831 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f870c0a29a0 0x7f870c0a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8728107180 tx=0x7f871401a040 comp rx=0 tx=0).stop 2024-03-17T23:55:57.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.779+0000 7f872e200700 1 -- 172.21.15.12:0/135859831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8728069f80 msgr2=0x7f87281015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:55:57.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.779+0000 7f872e200700 1 --2- 172.21.15.12:0/135859831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8728069f80 0x7f87281015f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f871800e9f0 tx=0x7f871800ed00 comp rx=0 tx=0).stop 2024-03-17T23:55:57.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.779+0000 7f8727e00700 1 -- 172.21.15.12:0/135859831 reap_dead start 2024-03-17T23:55:57.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.779+0000 7f872e200700 1 -- 172.21.15.12:0/135859831 shutdown_connections 2024-03-17T23:55:57.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.779+0000 7f872e200700 1 -- 172.21.15.12:0/135859831 >> 172.21.15.12:0/135859831 conn(0x7f8728077f80 msgr2=0x7f8728100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:55:57.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.779+0000 7f872e200700 1 -- 172.21.15.12:0/135859831 shutdown_connections 2024-03-17T23:55:57.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:55:57.779+0000 7f872e200700 1 -- 172.21.15.12:0/135859831 wait complete. 2024-03-17T23:55:57.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:57 smithi079 ceph-mon[36241]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-17T23:55:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:57 smithi191 ceph-mon[36708]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-17T23:55:58.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:58 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:55:58.740 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:58 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:55:58.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:58 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:55:59.627 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:59 smithi012 ceph-mon[31541]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 255 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-17T23:55:59.627 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:55:59.627 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:55:59 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:55:59.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:59 smithi079 ceph-mon[36241]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 255 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-17T23:55:59.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:59 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:55:59.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:55:59 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:55:59.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:59 smithi191 ceph-mon[36708]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 255 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-17T23:55:59.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:59 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:55:59.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:55:59 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:00.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:00 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:00.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:00 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:00.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:00 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:00.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:00 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:00.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:00 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:00.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:00 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:01.640 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:01 smithi012 ceph-mon[31541]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s; 36 B/s, 1 keys/s, 2 objects/s recovering 2024-03-17T23:56:01.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:01 smithi079 ceph-mon[36241]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s; 36 B/s, 1 keys/s, 2 objects/s recovering 2024-03-17T23:56:01.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:01 smithi191 ceph-mon[36708]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s; 36 B/s, 1 keys/s, 2 objects/s recovering 2024-03-17T23:56:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.268+0000 7f6f30200700 1 -- 172.21.15.12:0/1914531117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f28069f80 msgr2=0x7f6f2806a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.268+0000 7f6f30200700 1 --2- 172.21.15.12:0/1914531117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f28069f80 0x7f6f2806a400 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6f14009920 tx=0x7f6f14009c30 comp rx=0 tx=0).stop 2024-03-17T23:56:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.268+0000 7f6f30200700 1 -- 172.21.15.12:0/1914531117 shutdown_connections 2024-03-17T23:56:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.268+0000 7f6f30200700 1 --2- 172.21.15.12:0/1914531117 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f2806a940 0x7f6f2810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.268+0000 7f6f30200700 1 --2- 172.21.15.12:0/1914531117 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f28108300 0x7f6f281086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.268+0000 7f6f30200700 1 --2- 172.21.15.12:0/1914531117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f28069f80 0x7f6f2806a400 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.268+0000 7f6f30200700 1 -- 172.21.15.12:0/1914531117 >> 172.21.15.12:0/1914531117 conn(0x7f6f28077f80 msgr2=0x7f6f28078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.268+0000 7f6f30200700 1 -- 172.21.15.12:0/1914531117 shutdown_connections 2024-03-17T23:56:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.268+0000 7f6f30200700 1 -- 172.21.15.12:0/1914531117 wait complete. 2024-03-17T23:56:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.269+0000 7f6f30200700 1 Processor -- start 2024-03-17T23:56:02.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.269+0000 7f6f30200700 1 -- start start 2024-03-17T23:56:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f30200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f28069f80 0x7f6f2819ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f30200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f2806a940 0x7f6f2819b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f30200700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f28108300 0x7f6f2819f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f30200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6f28112580 con 0x7f6f2806a940 2024-03-17T23:56:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f30200700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6f28112400 con 0x7f6f28108300 2024-03-17T23:56:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f30200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6f28112700 con 0x7f6f28069f80 2024-03-17T23:56:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f2d800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f28069f80 0x7f6f2819ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f2d800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f28069f80 0x7f6f2819ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:33838/0 (socket says 172.21.15.12:33838) 2024-03-17T23:56:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f2d800700 1 -- 172.21.15.12:0/4276096 learned_addr learned my addr 172.21.15.12:0/4276096 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f2ce00700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f2806a940 0x7f6f2819b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f2e200700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f28108300 0x7f6f2819f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:02.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f2d800700 1 -- 172.21.15.12:0/4276096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f28108300 msgr2=0x7f6f2819f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:02.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f2d800700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f28108300 0x7f6f2819f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f2d800700 1 -- 172.21.15.12:0/4276096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f2806a940 msgr2=0x7f6f2819b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:02.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f2d800700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f2806a940 0x7f6f2819b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.270+0000 7f6f2d800700 1 -- 172.21.15.12:0/4276096 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2819fd60 con 0x7f6f28069f80 2024-03-17T23:56:02.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.271+0000 7f6f2d800700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f28069f80 0x7f6f2819ad50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00e9c0 tx=0x7f6f1c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:02.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.271+0000 7f6f26000700 1 -- 172.21.15.12:0/4276096 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c004a30 con 0x7f6f28069f80 2024-03-17T23:56:02.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.271+0000 7f6f26000700 1 -- 172.21.15.12:0/4276096 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f1c00cca0 con 0x7f6f28069f80 2024-03-17T23:56:02.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.271+0000 7f6f30200700 1 -- 172.21.15.12:0/4276096 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f281a0050 con 0x7f6f28069f80 2024-03-17T23:56:02.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.271+0000 7f6f26000700 1 -- 172.21.15.12:0/4276096 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c005680 con 0x7f6f28069f80 2024-03-17T23:56:02.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.271+0000 7f6f30200700 1 -- 172.21.15.12:0/4276096 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f281a78d0 con 0x7f6f28069f80 2024-03-17T23:56:02.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.272+0000 7f6f30200700 1 -- 172.21.15.12:0/4276096 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6f2810de90 con 0x7f6f28069f80 2024-03-17T23:56:02.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.274+0000 7f6f26000700 1 -- 172.21.15.12:0/4276096 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6f1c00c7c0 con 0x7f6f28069f80 2024-03-17T23:56:02.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.276+0000 7f6f26000700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6f100a29a0 0x7f6f100a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.276+0000 7f6f26000700 1 -- 172.21.15.12:0/4276096 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f6f1c0cf0d0 con 0x7f6f28069f80 2024-03-17T23:56:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.277+0000 7f6f2ce00700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6f100a29a0 0x7f6f100a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.277+0000 7f6f26000700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993] conn(0x7f6f100adb00 0x7f6f100affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.277+0000 7f6f26000700 1 -- 172.21.15.12:0/4276096 --> [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f1c01aec0 con 0x7f6f100adb00 2024-03-17T23:56:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.277+0000 7f6f26000700 1 -- 172.21.15.12:0/4276096 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f6f1c0cf480 con 0x7f6f28069f80 2024-03-17T23:56:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.277+0000 7f6f2e200700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993] conn(0x7f6f100adb00 0x7f6f100affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.277+0000 7f6f2ce00700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6f100a29a0 0x7f6f100a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400c010 tx=0x7f6f14019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.278+0000 7f6f2e200700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993] conn(0x7f6f100adb00 0x7f6f100affa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:02.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.279+0000 7f6f26000700 1 -- 172.21.15.12:0/4276096 <== osd.2 v2:172.21.15.191:6800/936135993 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f1c01aec0 con 0x7f6f100adb00 2024-03-17T23:56:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.298+0000 7f6f30200700 1 -- 172.21.15.12:0/4276096 --> [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6f28103990 con 0x7f6f100adb00 2024-03-17T23:56:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.298+0000 7f6f26000700 1 -- 172.21.15.12:0/4276096 <== osd.2 v2:172.21.15.191:6800/936135993 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6f28103990 con 0x7f6f100adb00 2024-03-17T23:56:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.298+0000 7f6f30200700 1 -- 172.21.15.12:0/4276096 >> [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993] conn(0x7f6f100adb00 msgr2=0x7f6f100affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.298+0000 7f6f30200700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993] conn(0x7f6f100adb00 0x7f6f100affa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.299+0000 7f6f30200700 1 -- 172.21.15.12:0/4276096 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6f100a29a0 msgr2=0x7f6f100a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.299+0000 7f6f30200700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6f100a29a0 0x7f6f100a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400c010 tx=0x7f6f14019040 comp rx=0 tx=0).stop 2024-03-17T23:56:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.299+0000 7f6f30200700 1 -- 172.21.15.12:0/4276096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f28069f80 msgr2=0x7f6f2819ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.299+0000 7f6f30200700 1 --2- 172.21.15.12:0/4276096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f28069f80 0x7f6f2819ad50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00e9c0 tx=0x7f6f1c00ecd0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.299+0000 7f6f2e200700 1 -- 172.21.15.12:0/4276096 reap_dead start 2024-03-17T23:56:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.299+0000 7f6f30200700 1 -- 172.21.15.12:0/4276096 shutdown_connections 2024-03-17T23:56:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.299+0000 7f6f30200700 1 -- 172.21.15.12:0/4276096 >> 172.21.15.12:0/4276096 conn(0x7f6f28077f80 msgr2=0x7f6f2810d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.299+0000 7f6f30200700 1 -- 172.21.15.12:0/4276096 shutdown_connections 2024-03-17T23:56:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.299+0000 7f6f30200700 1 -- 172.21.15.12:0/4276096 wait complete. 2024-03-17T23:56:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.399+0000 7fb581800700 1 -- 172.21.15.12:0/4224349103 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb57c06a920 msgr2=0x7fb57c10f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.399+0000 7fb581800700 1 --2- 172.21.15.12:0/4224349103 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb57c06a920 0x7fb57c10f160 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb57400b0d0 tx=0x7fb57400b3e0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.399+0000 7fb581800700 1 -- 172.21.15.12:0/4224349103 shutdown_connections 2024-03-17T23:56:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.399+0000 7fb581800700 1 --2- 172.21.15.12:0/4224349103 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb57c06a920 0x7fb57c10f160 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.399+0000 7fb581800700 1 --2- 172.21.15.12:0/4224349103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb57c108320 0x7fb57c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.399+0000 7fb581800700 1 --2- 172.21.15.12:0/4224349103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb57c069f80 0x7fb57c06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.399+0000 7fb581800700 1 -- 172.21.15.12:0/4224349103 >> 172.21.15.12:0/4224349103 conn(0x7fb57c078100 msgr2=0x7fb57c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.399+0000 7fb581800700 1 -- 172.21.15.12:0/4224349103 shutdown_connections 2024-03-17T23:56:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.399+0000 7fb581800700 1 -- 172.21.15.12:0/4224349103 wait complete. 2024-03-17T23:56:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.400+0000 7fb581800700 1 Processor -- start 2024-03-17T23:56:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.400+0000 7fb581800700 1 -- start start 2024-03-17T23:56:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.401+0000 7fb581800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb57c069f80 0x7fb57c19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.401+0000 7fb581800700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb57c06a920 0x7fb57c19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.401+0000 7fb581800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb57c108320 0x7fb57c198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.401+0000 7fb581800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c1125c0 con 0x7fb57c108320 2024-03-17T23:56:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.401+0000 7fb581800700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c112440 con 0x7fb57c06a920 2024-03-17T23:56:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.401+0000 7fb581800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c112740 con 0x7fb57c069f80 2024-03-17T23:56:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.401+0000 7fb57aa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb57c069f80 0x7fb57c19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.401+0000 7fb57a000700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb57c06a920 0x7fb57c19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.401+0000 7fb57aa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb57c069f80 0x7fb57c19e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:33864/0 (socket says 172.21.15.12:33864) 2024-03-17T23:56:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.401+0000 7fb57aa00700 1 -- 172.21.15.12:0/2177963563 learned_addr learned my addr 172.21.15.12:0/2177963563 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.401+0000 7fb57b400700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb57c108320 0x7fb57c198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.402+0000 7fb57aa00700 1 -- 172.21.15.12:0/2177963563 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb57c06a920 msgr2=0x7fb57c19ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.402+0000 7fb57aa00700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb57c06a920 0x7fb57c19ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.402+0000 7fb57aa00700 1 -- 172.21.15.12:0/2177963563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb57c108320 msgr2=0x7fb57c198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.402+0000 7fb57aa00700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb57c108320 0x7fb57c198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.402+0000 7fb57aa00700 1 -- 172.21.15.12:0/2177963563 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57c199080 con 0x7fb57c069f80 2024-03-17T23:56:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.402+0000 7fb57a000700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb57c06a920 0x7fb57c19ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.402+0000 7fb57b400700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb57c108320 0x7fb57c198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:56:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.402+0000 7fb57aa00700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb57c069f80 0x7fb57c19e710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00e9a0 tx=0x7fb56c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.402+0000 7fb573400700 1 -- 172.21.15.12:0/2177963563 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c00cbd0 con 0x7fb57c069f80 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.402+0000 7fb573400700 1 -- 172.21.15.12:0/2177963563 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb56c00cd70 con 0x7fb57c069f80 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.402+0000 7fb581800700 1 -- 172.21.15.12:0/2177963563 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb57c199370 con 0x7fb57c069f80 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.402+0000 7fb573400700 1 -- 172.21.15.12:0/2177963563 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c018750 con 0x7fb57c069f80 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.402+0000 7fb581800700 1 -- 172.21.15.12:0/2177963563 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb57c1a55e0 con 0x7fb57c069f80 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.403+0000 7fb581800700 1 -- 172.21.15.12:0/2177963563 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb57c0629a0 con 0x7fb57c069f80 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.406+0000 7fb573400700 1 -- 172.21.15.12:0/2177963563 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb56c010430 con 0x7fb57c069f80 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.407+0000 7fb573400700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb5600a2920 0x7fb5600a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.407+0000 7fb573400700 1 -- 172.21.15.12:0/2177963563 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fb56c0cf1b0 con 0x7fb57c069f80 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.407+0000 7fb57a000700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb5600a2920 0x7fb5600a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.407+0000 7fb573400700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857] conn(0x7fb5600ada80 0x7fb5600aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.408+0000 7fb573400700 1 -- 172.21.15.12:0/2177963563 --> [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb56c012ec0 con 0x7fb5600ada80 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.408+0000 7fb573400700 1 -- 172.21.15.12:0/2177963563 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fb56c0d6050 con 0x7fb57c069f80 2024-03-17T23:56:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.408+0000 7fb57b400700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857] conn(0x7fb5600ada80 0x7fb5600aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.408+0000 7fb57a000700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb5600a2920 0x7fb5600a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb57c0786e0 tx=0x7fb568009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.408+0000 7fb57b400700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857] conn(0x7fb5600ada80 0x7fb5600aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.410+0000 7fb573400700 1 -- 172.21.15.12:0/2177963563 <== osd.1 v2:172.21.15.79:6800/890307857 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb56c012ec0 con 0x7fb5600ada80 2024-03-17T23:56:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.428+0000 7fb581800700 1 -- 172.21.15.12:0/2177963563 --> [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb57c062dd0 con 0x7fb5600ada80 2024-03-17T23:56:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.429+0000 7fb573400700 1 -- 172.21.15.12:0/2177963563 <== osd.1 v2:172.21.15.79:6800/890307857 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb57c062dd0 con 0x7fb5600ada80 2024-03-17T23:56:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.429+0000 7fb581800700 1 -- 172.21.15.12:0/2177963563 >> [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857] conn(0x7fb5600ada80 msgr2=0x7fb5600aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.429+0000 7fb581800700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857] conn(0x7fb5600ada80 0x7fb5600aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.429+0000 7fb581800700 1 -- 172.21.15.12:0/2177963563 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb5600a2920 msgr2=0x7fb5600a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.429+0000 7fb581800700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb5600a2920 0x7fb5600a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb57c0786e0 tx=0x7fb568009260 comp rx=0 tx=0).stop 2024-03-17T23:56:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.429+0000 7fb581800700 1 -- 172.21.15.12:0/2177963563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb57c069f80 msgr2=0x7fb57c19e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.429+0000 7fb581800700 1 --2- 172.21.15.12:0/2177963563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb57c069f80 0x7fb57c19e710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00e9a0 tx=0x7fb56c00ecb0 comp rx=0 tx=0).stop 2024-03-17T23:56:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.429+0000 7fb57b400700 1 -- 172.21.15.12:0/2177963563 reap_dead start 2024-03-17T23:56:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.430+0000 7fb581800700 1 -- 172.21.15.12:0/2177963563 shutdown_connections 2024-03-17T23:56:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.430+0000 7fb581800700 1 -- 172.21.15.12:0/2177963563 >> 172.21.15.12:0/2177963563 conn(0x7fb57c078100 msgr2=0x7fb57c0ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.430+0000 7fb581800700 1 -- 172.21.15.12:0/2177963563 shutdown_connections 2024-03-17T23:56:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:02.430+0000 7fb581800700 1 -- 172.21.15.12:0/2177963563 wait complete. 2024-03-17T23:56:03.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.295+0000 7fd68fe00700 1 -- 172.21.15.12:0/3613840050 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd690103ca0 msgr2=0x7fd69010c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.295+0000 7fd68fe00700 1 --2- 172.21.15.12:0/3613840050 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd690103ca0 0x7fd69010c190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009990 tx=0x7fd67c009ca0 comp rx=0 tx=0).stop 2024-03-17T23:56:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.295+0000 7fd68fe00700 1 -- 172.21.15.12:0/3613840050 shutdown_connections 2024-03-17T23:56:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.295+0000 7fd68fe00700 1 --2- 172.21.15.12:0/3613840050 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd69010c7b0 0x7fd69010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.295+0000 7fd68fe00700 1 --2- 172.21.15.12:0/3613840050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd690103380 0x7fd690103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.295+0000 7fd68fe00700 1 --2- 172.21.15.12:0/3613840050 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd690103ca0 0x7fd69010c190 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.295+0000 7fd68fe00700 1 -- 172.21.15.12:0/3613840050 >> 172.21.15.12:0/3613840050 conn(0x7fd6900fcb90 msgr2=0x7fd6900fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.296+0000 7fd68fe00700 1 -- 172.21.15.12:0/3613840050 shutdown_connections 2024-03-17T23:56:03.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.296+0000 7fd68fe00700 1 -- 172.21.15.12:0/3613840050 wait complete. 2024-03-17T23:56:03.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.297+0000 7fd68fe00700 1 Processor -- start 2024-03-17T23:56:03.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.297+0000 7fd68fe00700 1 -- start start 2024-03-17T23:56:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.297+0000 7fd68fe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd690103380 0x7fd690100fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.297+0000 7fd68fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd690103ca0 0x7fd690101520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.297+0000 7fd68fe00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd69010c7b0 0x7fd690101a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.297+0000 7fd68fe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd690112550 con 0x7fd690103ca0 2024-03-17T23:56:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.297+0000 7fd68fe00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd6901123d0 con 0x7fd69010c7b0 2024-03-17T23:56:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.297+0000 7fd68fe00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd6901126d0 con 0x7fd690103380 2024-03-17T23:56:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.298+0000 7fd68ea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd690103380 0x7fd690100fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.298+0000 7fd68ea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd690103380 0x7fd690100fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:33876/0 (socket says 172.21.15.12:33876) 2024-03-17T23:56:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.298+0000 7fd68ea00700 1 -- 172.21.15.12:0/2206573553 learned_addr learned my addr 172.21.15.12:0/2206573553 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.298+0000 7fd68e000700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd690103ca0 0x7fd690101520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.298+0000 7fd68f400700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd69010c7b0 0x7fd690101a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.298+0000 7fd68ea00700 1 -- 172.21.15.12:0/2206573553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd69010c7b0 msgr2=0x7fd690101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.298+0000 7fd68ea00700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd69010c7b0 0x7fd690101a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.298+0000 7fd68ea00700 1 -- 172.21.15.12:0/2206573553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd690103ca0 msgr2=0x7fd690101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.298+0000 7fd68ea00700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd690103ca0 0x7fd690101520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.298+0000 7fd68ea00700 1 -- 172.21.15.12:0/2206573553 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6901a6f30 con 0x7fd690103380 2024-03-17T23:56:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.299+0000 7fd68ea00700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd690103380 0x7fd690100fe0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd68000d980 tx=0x7fd68000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.299+0000 7fd687400700 1 -- 172.21.15.12:0/2206573553 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd680016070 con 0x7fd690103380 2024-03-17T23:56:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.299+0000 7fd687400700 1 -- 172.21.15.12:0/2206573553 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd680004d10 con 0x7fd690103380 2024-03-17T23:56:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.299+0000 7fd68fe00700 1 -- 172.21.15.12:0/2206573553 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6901a7220 con 0x7fd690103380 2024-03-17T23:56:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.299+0000 7fd68fe00700 1 -- 172.21.15.12:0/2206573553 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6901a7730 con 0x7fd690103380 2024-03-17T23:56:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.299+0000 7fd687400700 1 -- 172.21.15.12:0/2206573553 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd680005670 con 0x7fd690103380 2024-03-17T23:56:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.300+0000 7fd68fe00700 1 -- 172.21.15.12:0/2206573553 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd670001030 con 0x7fd690103380 2024-03-17T23:56:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.302+0000 7fd687400700 1 -- 172.21.15.12:0/2206573553 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd680004220 con 0x7fd690103380 2024-03-17T23:56:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.304+0000 7fd687400700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd6740a2920 0x7fd6740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.304+0000 7fd687400700 1 -- 172.21.15.12:0/2206573553 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fd6800cf350 con 0x7fd690103380 2024-03-17T23:56:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.304+0000 7fd68e000700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd6740a2920 0x7fd6740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.304+0000 7fd687400700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575] conn(0x7fd6740ada80 0x7fd6740aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.304+0000 7fd687400700 1 -- 172.21.15.12:0/2206573553 --> [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd68001be50 con 0x7fd6740ada80 2024-03-17T23:56:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.304+0000 7fd687400700 1 -- 172.21.15.12:0/2206573553 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fd6800cf700 con 0x7fd690103380 2024-03-17T23:56:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.305+0000 7fd68f400700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575] conn(0x7fd6740ada80 0x7fd6740aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.305+0000 7fd68e000700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd6740a2920 0x7fd6740a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd690102a70 tx=0x7fd67c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.305+0000 7fd68f400700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575] conn(0x7fd6740ada80 0x7fd6740aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.306+0000 7fd687400700 1 -- 172.21.15.12:0/2206573553 <== osd.3 v2:172.21.15.12:6810/769150575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd68001be50 con 0x7fd6740ada80 2024-03-17T23:56:03.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.325+0000 7fd68fe00700 1 -- 172.21.15.12:0/2206573553 --> [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd670002e60 con 0x7fd6740ada80 2024-03-17T23:56:03.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.326+0000 7fd687400700 1 -- 172.21.15.12:0/2206573553 <== osd.3 v2:172.21.15.12:6810/769150575 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd670002e60 con 0x7fd6740ada80 2024-03-17T23:56:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.326+0000 7fd68fe00700 1 -- 172.21.15.12:0/2206573553 >> [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575] conn(0x7fd6740ada80 msgr2=0x7fd6740aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.326+0000 7fd68fe00700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575] conn(0x7fd6740ada80 0x7fd6740aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.327+0000 7fd68fe00700 1 -- 172.21.15.12:0/2206573553 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd6740a2920 msgr2=0x7fd6740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.327+0000 7fd68fe00700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd6740a2920 0x7fd6740a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd690102a70 tx=0x7fd67c00b540 comp rx=0 tx=0).stop 2024-03-17T23:56:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.327+0000 7fd68fe00700 1 -- 172.21.15.12:0/2206573553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd690103380 msgr2=0x7fd690100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.327+0000 7fd68fe00700 1 --2- 172.21.15.12:0/2206573553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd690103380 0x7fd690100fe0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd68000d980 tx=0x7fd68000dc90 comp rx=0 tx=0).stop 2024-03-17T23:56:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.327+0000 7fd68f400700 1 -- 172.21.15.12:0/2206573553 reap_dead start 2024-03-17T23:56:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.327+0000 7fd68fe00700 1 -- 172.21.15.12:0/2206573553 shutdown_connections 2024-03-17T23:56:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.327+0000 7fd68fe00700 1 -- 172.21.15.12:0/2206573553 >> 172.21.15.12:0/2206573553 conn(0x7fd6900fcb90 msgr2=0x7fd6900fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.327+0000 7fd68fe00700 1 -- 172.21.15.12:0/2206573553 shutdown_connections 2024-03-17T23:56:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:03.327+0000 7fd68fe00700 1 -- 172.21.15.12:0/2206573553 wait complete. 2024-03-17T23:56:03.793 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:03 smithi012 ceph-mon[31541]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s; 36 B/s, 1 keys/s, 2 objects/s recovering 2024-03-17T23:56:03.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:03 smithi079 ceph-mon[36241]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s; 36 B/s, 1 keys/s, 2 objects/s recovering 2024-03-17T23:56:03.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:03 smithi191 ceph-mon[36708]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s; 36 B/s, 1 keys/s, 2 objects/s recovering 2024-03-17T23:56:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.226+0000 7f977f200700 1 -- 172.21.15.12:0/2710999702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f977806a920 msgr2=0x7f977810f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.226+0000 7f977f200700 1 --2- 172.21.15.12:0/2710999702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f977806a920 0x7f977810f160 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9770009fc0 tx=0x7f977000b330 comp rx=0 tx=0).stop 2024-03-17T23:56:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.227+0000 7f977f200700 1 -- 172.21.15.12:0/2710999702 shutdown_connections 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.227+0000 7f977f200700 1 --2- 172.21.15.12:0/2710999702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f977806a920 0x7f977810f160 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.227+0000 7f977f200700 1 --2- 172.21.15.12:0/2710999702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9778108320 0x7f9778108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.227+0000 7f977f200700 1 --2- 172.21.15.12:0/2710999702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9778069f80 0x7f977806a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.227+0000 7f977f200700 1 -- 172.21.15.12:0/2710999702 >> 172.21.15.12:0/2710999702 conn(0x7f9778078100 msgr2=0x7f9778078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.227+0000 7f977f200700 1 -- 172.21.15.12:0/2710999702 shutdown_connections 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.227+0000 7f977f200700 1 -- 172.21.15.12:0/2710999702 wait complete. 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.228+0000 7f977f200700 1 Processor -- start 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.228+0000 7f977f200700 1 -- start start 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f977f200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9778069f80 0x7f97781a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f977f200700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f977806a920 0x7f97781a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f977f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9778108320 0x7f977819aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f977f200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f97781125c0 con 0x7f9778108320 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f977f200700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9778112440 con 0x7f977806a920 2024-03-17T23:56:05.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f977f200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9778112740 con 0x7f9778069f80 2024-03-17T23:56:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f976fe00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f977806a920 0x7f97781a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f9777e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9778069f80 0x7f97781a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f976fe00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f977806a920 0x7f97781a0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:54972/0 (socket says 172.21.15.12:54972) 2024-03-17T23:56:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f976fe00700 1 -- 172.21.15.12:0/3793996164 learned_addr learned my addr 172.21.15.12:0/3793996164 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f977d200700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9778108320 0x7f977819aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f976fe00700 1 -- 172.21.15.12:0/3793996164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9778069f80 msgr2=0x7f97781a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f976fe00700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9778069f80 0x7f97781a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f976fe00700 1 -- 172.21.15.12:0/3793996164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9778108320 msgr2=0x7f977819aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.229+0000 7f976fe00700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9778108320 0x7f977819aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.230+0000 7f976fe00700 1 -- 172.21.15.12:0/3793996164 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977819b240 con 0x7f977806a920 2024-03-17T23:56:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.230+0000 7f9777e00700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9778069f80 0x7f97781a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.230+0000 7f976fe00700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f977806a920 0x7f97781a0e60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f976000ec10 tx=0x7f976000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:05.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.230+0000 7f9775600700 1 -- 172.21.15.12:0/3793996164 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976000cd40 con 0x7f977806a920 2024-03-17T23:56:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.230+0000 7f9775600700 1 -- 172.21.15.12:0/3793996164 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9760004500 con 0x7f977806a920 2024-03-17T23:56:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.230+0000 7f9775600700 1 -- 172.21.15.12:0/3793996164 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9760010690 con 0x7f977806a920 2024-03-17T23:56:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.230+0000 7f977f200700 1 -- 172.21.15.12:0/3793996164 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f977819b530 con 0x7f977806a920 2024-03-17T23:56:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.230+0000 7f977f200700 1 -- 172.21.15.12:0/3793996164 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97781a77f0 con 0x7f977806a920 2024-03-17T23:56:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.231+0000 7f977f200700 1 -- 172.21.15.12:0/3793996164 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f97781005c0 con 0x7f977806a920 2024-03-17T23:56:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.233+0000 7f9775600700 1 -- 172.21.15.12:0/3793996164 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9760003f10 con 0x7f977806a920 2024-03-17T23:56:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.235+0000 7f9775600700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f975c0a29a0 0x7f975c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.235+0000 7f9775600700 1 -- 172.21.15.12:0/3793996164 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f97600cf3b0 con 0x7f977806a920 2024-03-17T23:56:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.235+0000 7f9777e00700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f975c0a29a0 0x7f975c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.235+0000 7f9775600700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703] conn(0x7f975c0adb00 0x7f975c0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.235+0000 7f9775600700 1 -- 172.21.15.12:0/3793996164 --> [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f975c0b0670 con 0x7f975c0adb00 2024-03-17T23:56:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.235+0000 7f9775600700 1 -- 172.21.15.12:0/3793996164 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f97600cf800 con 0x7f977806a920 2024-03-17T23:56:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.236+0000 7f977d200700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703] conn(0x7f975c0adb00 0x7f975c0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.236+0000 7f9777e00700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f975c0a29a0 0x7f975c0a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9768009fd0 tx=0x7f9768009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.236+0000 7f977d200700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703] conn(0x7f975c0adb00 0x7f975c0affa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.237+0000 7f9775600700 1 -- 172.21.15.12:0/3793996164 <== osd.8 v2:172.21.15.191:6816/704430703 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f975c0b0670 con 0x7f975c0adb00 2024-03-17T23:56:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.266+0000 7f977f200700 1 -- 172.21.15.12:0/3793996164 --> [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f97781039d0 con 0x7f975c0adb00 2024-03-17T23:56:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.267+0000 7f9775600700 1 -- 172.21.15.12:0/3793996164 <== osd.8 v2:172.21.15.191:6816/704430703 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f97781039d0 con 0x7f975c0adb00 2024-03-17T23:56:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.267+0000 7f977f200700 1 -- 172.21.15.12:0/3793996164 >> [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703] conn(0x7f975c0adb00 msgr2=0x7f975c0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.267+0000 7f977f200700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703] conn(0x7f975c0adb00 0x7f975c0affa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.268+0000 7f977f200700 1 -- 172.21.15.12:0/3793996164 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f975c0a29a0 msgr2=0x7f975c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.268+0000 7f977f200700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f975c0a29a0 0x7f975c0a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9768009fd0 tx=0x7f9768009260 comp rx=0 tx=0).stop 2024-03-17T23:56:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.268+0000 7f977f200700 1 -- 172.21.15.12:0/3793996164 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f977806a920 msgr2=0x7f97781a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.268+0000 7f977f200700 1 --2- 172.21.15.12:0/3793996164 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f977806a920 0x7f97781a0e60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f976000ec10 tx=0x7f976000ef20 comp rx=0 tx=0).stop 2024-03-17T23:56:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.268+0000 7f977d200700 1 -- 172.21.15.12:0/3793996164 reap_dead start 2024-03-17T23:56:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.268+0000 7f977f200700 1 -- 172.21.15.12:0/3793996164 shutdown_connections 2024-03-17T23:56:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.268+0000 7f977f200700 1 -- 172.21.15.12:0/3793996164 >> 172.21.15.12:0/3793996164 conn(0x7f9778078100 msgr2=0x7f97780ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.268+0000 7f977f200700 1 -- 172.21.15.12:0/3793996164 shutdown_connections 2024-03-17T23:56:05.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:05.268+0000 7f977f200700 1 -- 172.21.15.12:0/3793996164 wait complete. 2024-03-17T23:56:05.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:05 smithi079 ceph-mon[36241]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 211 B/s wr, 9 op/s; 30 B/s, 1 keys/s, 2 objects/s recovering 2024-03-17T23:56:05.808 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:05 smithi012 ceph-mon[31541]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 211 B/s wr, 9 op/s; 30 B/s, 1 keys/s, 2 objects/s recovering 2024-03-17T23:56:05.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:05 smithi191 ceph-mon[36708]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 211 B/s wr, 9 op/s; 30 B/s, 1 keys/s, 2 objects/s recovering 2024-03-17T23:56:06.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.329+0000 7fa977e00700 1 -- 172.21.15.12:0/413002208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97810c7a0 msgr2=0x7fa97810ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:06.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.329+0000 7fa977e00700 1 --2- 172.21.15.12:0/413002208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97810c7a0 0x7fa97810ec60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa97000b0a0 tx=0x7fa97000b3b0 comp rx=0 tx=0).stop 2024-03-17T23:56:06.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.329+0000 7fa977e00700 1 -- 172.21.15.12:0/413002208 shutdown_connections 2024-03-17T23:56:06.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.329+0000 7fa977e00700 1 --2- 172.21.15.12:0/413002208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97810c7a0 0x7fa97810ec60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:06.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.329+0000 7fa977e00700 1 --2- 172.21.15.12:0/413002208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa978103370 0x7fa978103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:06.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.329+0000 7fa977e00700 1 --2- 172.21.15.12:0/413002208 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa978103c90 0x7fa97810c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:06.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.329+0000 7fa977e00700 1 -- 172.21.15.12:0/413002208 >> 172.21.15.12:0/413002208 conn(0x7fa9780fcba0 msgr2=0x7fa9780fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:06.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.330+0000 7fa977e00700 1 -- 172.21.15.12:0/413002208 shutdown_connections 2024-03-17T23:56:06.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.330+0000 7fa977e00700 1 -- 172.21.15.12:0/413002208 wait complete. 2024-03-17T23:56:06.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.331+0000 7fa977e00700 1 Processor -- start 2024-03-17T23:56:06.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.331+0000 7fa977e00700 1 -- start start 2024-03-17T23:56:06.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.331+0000 7fa977e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa978103370 0x7fa978100fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:06.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.331+0000 7fa977e00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa978103c90 0x7fa978101510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:06.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.331+0000 7fa977e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97810c7a0 0x7fa978101a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.331+0000 7fa977e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa978112540 con 0x7fa978103370 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.331+0000 7fa977e00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa9781123c0 con 0x7fa978103c90 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.331+0000 7fa977e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa9781126c0 con 0x7fa97810c7a0 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.332+0000 7fa976a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa978103370 0x7fa978100fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.332+0000 7fa976000700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa978103c90 0x7fa978101510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.332+0000 7fa976a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa978103370 0x7fa978100fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50314/0 (socket says 172.21.15.12:50314) 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.332+0000 7fa976a00700 1 -- 172.21.15.12:0/1302193191 learned_addr learned my addr 172.21.15.12:0/1302193191 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.332+0000 7fa977400700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97810c7a0 0x7fa978101a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.332+0000 7fa977400700 1 -- 172.21.15.12:0/1302193191 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa978103c90 msgr2=0x7fa978101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.332+0000 7fa977400700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa978103c90 0x7fa978101510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.332+0000 7fa977400700 1 -- 172.21.15.12:0/1302193191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa978103370 msgr2=0x7fa978100fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.332+0000 7fa977400700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa978103370 0x7fa978100fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.332+0000 7fa977400700 1 -- 172.21.15.12:0/1302193191 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9781a6e80 con 0x7fa97810c7a0 2024-03-17T23:56:06.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.332+0000 7fa976000700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa978103c90 0x7fa978101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.332+0000 7fa977400700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97810c7a0 0x7fa978101a50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa97000b070 tx=0x7fa970004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.333+0000 7fa96f400700 1 -- 172.21.15.12:0/1302193191 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970021070 con 0x7fa97810c7a0 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.333+0000 7fa96f400700 1 -- 172.21.15.12:0/1302193191 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa970007b80 con 0x7fa97810c7a0 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.333+0000 7fa96f400700 1 -- 172.21.15.12:0/1302193191 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970005040 con 0x7fa97810c7a0 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.333+0000 7fa977e00700 1 -- 172.21.15.12:0/1302193191 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9781a7110 con 0x7fa97810c7a0 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.333+0000 7fa977e00700 1 -- 172.21.15.12:0/1302193191 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9781a76a0 con 0x7fa97810c7a0 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.333+0000 7fa977e00700 1 -- 172.21.15.12:0/1302193191 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa9780feab0 con 0x7fa97810c7a0 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.337+0000 7fa96f400700 1 -- 172.21.15.12:0/1302193191 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa970024070 con 0x7fa97810c7a0 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.338+0000 7fa96f400700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa95c0a29a0 0x7fa95c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.338+0000 7fa96f400700 1 -- 172.21.15.12:0/1302193191 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fa97000f070 con 0x7fa97810c7a0 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.338+0000 7fa976a00700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa95c0a29a0 0x7fa95c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.339+0000 7fa96f400700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073] conn(0x7fa95c0adb00 0x7fa95c0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.339+0000 7fa96f400700 1 -- 172.21.15.12:0/1302193191 --> [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa97000aea0 con 0x7fa95c0adb00 2024-03-17T23:56:06.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.339+0000 7fa96f400700 1 -- 172.21.15.12:0/1302193191 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fa9700d07c0 con 0x7fa97810c7a0 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.339+0000 7fa976000700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073] conn(0x7fa95c0adb00 0x7fa95c0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.339+0000 7fa976a00700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa95c0a29a0 0x7fa95c0a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa968009d20 tx=0x7fa968009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.340+0000 7fa976000700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073] conn(0x7fa95c0adb00 0x7fa95c0affa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.341+0000 7fa96f400700 1 -- 172.21.15.12:0/1302193191 <== osd.6 v2:172.21.15.12:6818/3661372073 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa97000aea0 con 0x7fa95c0adb00 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.367+0000 7fa977e00700 1 -- 172.21.15.12:0/1302193191 --> [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa97806a860 con 0x7fa95c0adb00 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.368+0000 7fa96f400700 1 -- 172.21.15.12:0/1302193191 <== osd.6 v2:172.21.15.12:6818/3661372073 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa97806a860 con 0x7fa95c0adb00 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.369+0000 7fa977e00700 1 -- 172.21.15.12:0/1302193191 >> [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073] conn(0x7fa95c0adb00 msgr2=0x7fa95c0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.369+0000 7fa977e00700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073] conn(0x7fa95c0adb00 0x7fa95c0affa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.369+0000 7fa977e00700 1 -- 172.21.15.12:0/1302193191 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa95c0a29a0 msgr2=0x7fa95c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.369+0000 7fa977e00700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa95c0a29a0 0x7fa95c0a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa968009d20 tx=0x7fa968009260 comp rx=0 tx=0).stop 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.369+0000 7fa977e00700 1 -- 172.21.15.12:0/1302193191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97810c7a0 msgr2=0x7fa978101a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.369+0000 7fa977e00700 1 --2- 172.21.15.12:0/1302193191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97810c7a0 0x7fa978101a50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa97000b070 tx=0x7fa970004c80 comp rx=0 tx=0).stop 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.369+0000 7fa977400700 1 -- 172.21.15.12:0/1302193191 reap_dead start 2024-03-17T23:56:06.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.369+0000 7fa977e00700 1 -- 172.21.15.12:0/1302193191 shutdown_connections 2024-03-17T23:56:06.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.369+0000 7fa977e00700 1 -- 172.21.15.12:0/1302193191 >> 172.21.15.12:0/1302193191 conn(0x7fa9780fcba0 msgr2=0x7fa97810ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:06.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.369+0000 7fa977e00700 1 -- 172.21.15.12:0/1302193191 shutdown_connections 2024-03-17T23:56:06.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.369+0000 7fa977e00700 1 -- 172.21.15.12:0/1302193191 wait complete. 2024-03-17T23:56:06.904 INFO:teuthology.orchestra.run.smithi012.stdout:55834574932 2024-03-17T23:56:06.904 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-17T23:56:06.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.009+0000 7fd76a000700 1 -- 172.21.15.12:0/3729704234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7640745f0 msgr2=0x7fd7640749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:06.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.009+0000 7fd76a000700 1 --2- 172.21.15.12:0/3729704234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7640745f0 0x7fd7640749d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd7540098f0 tx=0x7fd754009c00 comp rx=0 tx=0).stop 2024-03-17T23:56:06.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.009+0000 7fd76a000700 1 -- 172.21.15.12:0/3729704234 shutdown_connections 2024-03-17T23:56:06.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.009+0000 7fd76a000700 1 --2- 172.21.15.12:0/3729704234 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd76410ebb0 0x7fd764110fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:06.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.009+0000 7fd76a000700 1 --2- 172.21.15.12:0/3729704234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7640745f0 0x7fd7640749d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.009+0000 7fd76a000700 1 --2- 172.21.15.12:0/3729704234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd764074f10 0x7fd76410e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.009+0000 7fd76a000700 1 -- 172.21.15.12:0/3729704234 >> 172.21.15.12:0/3729704234 conn(0x7fd7640fdda0 msgr2=0x7fd7641001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.009+0000 7fd76a000700 1 -- 172.21.15.12:0/3729704234 shutdown_connections 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.009+0000 7fd76a000700 1 -- 172.21.15.12:0/3729704234 wait complete. 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.011+0000 7fd76a000700 1 Processor -- start 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.011+0000 7fd76a000700 1 -- start start 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.011+0000 7fd76a000700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7640745f0 0x7fd76419ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.011+0000 7fd76a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd764074f10 0x7fd76419b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.011+0000 7fd76a000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd76410ebb0 0x7fd76419f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.011+0000 7fd76a000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd764114840 con 0x7fd764074f10 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.011+0000 7fd76a000700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd7641146c0 con 0x7fd7640745f0 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.011+0000 7fd76a000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd7641149c0 con 0x7fd76410ebb0 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.012+0000 7fd763400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7640745f0 0x7fd76419ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:06.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.012+0000 7fd763400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7640745f0 0x7fd76419ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:54984/0 (socket says 172.21.15.12:54984) 2024-03-17T23:56:06.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.012+0000 7fd763400700 1 -- 172.21.15.12:0/240713726 learned_addr learned my addr 172.21.15.12:0/240713726 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:06.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.012+0000 7fd763e00700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd76410ebb0 0x7fd76419f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:06.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.012+0000 7fd762a00700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd764074f10 0x7fd76419b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:06.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.012+0000 7fd763400700 1 -- 172.21.15.12:0/240713726 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd76410ebb0 msgr2=0x7fd76419f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:06.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.012+0000 7fd763400700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd76410ebb0 0x7fd76419f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:06.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.012+0000 7fd763400700 1 -- 172.21.15.12:0/240713726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd764074f10 msgr2=0x7fd76419b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:06.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.012+0000 7fd763400700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd764074f10 0x7fd76419b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:06.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.012+0000 7fd763400700 1 -- 172.21.15.12:0/240713726 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76419fd50 con 0x7fd7640745f0 2024-03-17T23:56:06.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.013+0000 7fd763400700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7640745f0 0x7fd76419ad90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd754005950 tx=0x7fd75400fb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:06.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.013+0000 7fd75be00700 1 -- 172.21.15.12:0/240713726 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7540051f0 con 0x7fd7640745f0 2024-03-17T23:56:06.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.013+0000 7fd75be00700 1 -- 172.21.15.12:0/240713726 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd75400b560 con 0x7fd7640745f0 2024-03-17T23:56:06.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.013+0000 7fd75be00700 1 -- 172.21.15.12:0/240713726 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd754003ba0 con 0x7fd7640745f0 2024-03-17T23:56:06.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.013+0000 7fd76a000700 1 -- 172.21.15.12:0/240713726 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd76419ffe0 con 0x7fd7640745f0 2024-03-17T23:56:06.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.013+0000 7fd76a000700 1 -- 172.21.15.12:0/240713726 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7641a78c0 con 0x7fd7640745f0 2024-03-17T23:56:06.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.014+0000 7fd76a000700 1 -- 172.21.15.12:0/240713726 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd7640629e0 con 0x7fd7640745f0 2024-03-17T23:56:06.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.016+0000 7fd75be00700 1 -- 172.21.15.12:0/240713726 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd75400b710 con 0x7fd7640745f0 2024-03-17T23:56:06.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.019+0000 7fd75be00700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd7480a29a0 0x7fd7480a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:06.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.019+0000 7fd75be00700 1 -- 172.21.15.12:0/240713726 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fd7540d02d0 con 0x7fd7640745f0 2024-03-17T23:56:06.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.019+0000 7fd762a00700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd7480a29a0 0x7fd7480a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:06.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.019+0000 7fd75be00700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360] conn(0x7fd7480adb00 0x7fd7480affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:06.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.019+0000 7fd75be00700 1 -- 172.21.15.12:0/240713726 --> [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7480b0670 con 0x7fd7480adb00 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.019+0000 7fd75be00700 1 -- 172.21.15.12:0/240713726 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fd7540d0680 con 0x7fd7640745f0 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.019+0000 7fd763e00700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360] conn(0x7fd7480adb00 0x7fd7480affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.020+0000 7fd763e00700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360] conn(0x7fd7480adb00 0x7fd7480affa0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.020+0000 7fd762a00700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd7480a29a0 0x7fd7480a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd76410fe60 tx=0x7fd750009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.021+0000 7fd75be00700 1 -- 172.21.15.12:0/240713726 <== osd.4 v2:172.21.15.79:6808/3381761360 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd7480b0670 con 0x7fd7480adb00 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.045+0000 7fd76a000700 1 -- 172.21.15.12:0/240713726 --> [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd764062de0 con 0x7fd7480adb00 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.045+0000 7fd75be00700 1 -- 172.21.15.12:0/240713726 <== osd.4 v2:172.21.15.79:6808/3381761360 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd764062de0 con 0x7fd7480adb00 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.046+0000 7fd76a000700 1 -- 172.21.15.12:0/240713726 >> [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360] conn(0x7fd7480adb00 msgr2=0x7fd7480affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.046+0000 7fd76a000700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360] conn(0x7fd7480adb00 0x7fd7480affa0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.046+0000 7fd76a000700 1 -- 172.21.15.12:0/240713726 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd7480a29a0 msgr2=0x7fd7480a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.046+0000 7fd76a000700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd7480a29a0 0x7fd7480a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd76410fe60 tx=0x7fd750009260 comp rx=0 tx=0).stop 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.046+0000 7fd76a000700 1 -- 172.21.15.12:0/240713726 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7640745f0 msgr2=0x7fd76419ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.046+0000 7fd76a000700 1 --2- 172.21.15.12:0/240713726 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7640745f0 0x7fd76419ad90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd754005950 tx=0x7fd75400fb20 comp rx=0 tx=0).stop 2024-03-17T23:56:06.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.046+0000 7fd763e00700 1 -- 172.21.15.12:0/240713726 reap_dead start 2024-03-17T23:56:06.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.046+0000 7fd76a000700 1 -- 172.21.15.12:0/240713726 shutdown_connections 2024-03-17T23:56:06.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.046+0000 7fd76a000700 1 -- 172.21.15.12:0/240713726 >> 172.21.15.12:0/240713726 conn(0x7fd7640fdda0 msgr2=0x7fd7640feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:06.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.046+0000 7fd76a000700 1 -- 172.21.15.12:0/240713726 shutdown_connections 2024-03-17T23:56:06.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:06.046+0000 7fd76a000700 1 -- 172.21.15.12:0/240713726 wait complete. 2024-03-17T23:56:07.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:07 smithi012 ceph-mon[31541]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 192 B/s wr, 11 op/s; 27 B/s, 0 keys/s, 1 objects/s recovering 2024-03-17T23:56:07.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:07 smithi079 ceph-mon[36241]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 192 B/s wr, 11 op/s; 27 B/s, 0 keys/s, 1 objects/s recovering 2024-03-17T23:56:07.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:07 smithi191 ceph-mon[36708]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 192 B/s wr, 11 op/s; 27 B/s, 0 keys/s, 1 objects/s recovering 2024-03-17T23:56:08.435 INFO:teuthology.orchestra.run.smithi012.stdout:197568495672 2024-03-17T23:56:08.435 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-17T23:56:09.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:09 smithi012 ceph-mon[31541]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 24 B/s, 0 keys/s, 1 objects/s recovering 2024-03-17T23:56:09.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:09 smithi079 ceph-mon[36241]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 24 B/s, 0 keys/s, 1 objects/s recovering 2024-03-17T23:56:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:09 smithi191 ceph-mon[36708]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 24 B/s, 0 keys/s, 1 objects/s recovering 2024-03-17T23:56:10.076 INFO:teuthology.orchestra.run.smithi012.stdout:141733920837 2024-03-17T23:56:10.076 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-17T23:56:10.596 INFO:teuthology.orchestra.run.smithi012.stdout:253403070509 2024-03-17T23:56:10.596 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-17T23:56:10.929 INFO:teuthology.orchestra.run.smithi012.stdout:309237645343 2024-03-17T23:56:10.929 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-17T23:56:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.717+0000 7fd6b0a00700 1 -- 172.21.15.12:0/907857447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6ac104500 msgr2=0x7fd6ac104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.717+0000 7fd6b0a00700 1 --2- 172.21.15.12:0/907857447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6ac104500 0x7fd6ac104980 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd6980099c0 tx=0x7fd698009cd0 comp rx=0 tx=0).stop 2024-03-17T23:56:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.718+0000 7fd6b0a00700 1 -- 172.21.15.12:0/907857447 shutdown_connections 2024-03-17T23:56:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.718+0000 7fd6b0a00700 1 --2- 172.21.15.12:0/907857447 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6ac104ec0 0x7fd6ac111380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.718+0000 7fd6b0a00700 1 --2- 172.21.15.12:0/907857447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6ac10a5b0 0x7fd6ac10a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.718+0000 7fd6b0a00700 1 --2- 172.21.15.12:0/907857447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6ac104500 0x7fd6ac104980 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.718+0000 7fd6b0a00700 1 -- 172.21.15.12:0/907857447 >> 172.21.15.12:0/907857447 conn(0x7fd6ac0fffd0 msgr2=0x7fd6ac1023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.718+0000 7fd6b0a00700 1 -- 172.21.15.12:0/907857447 shutdown_connections 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.718+0000 7fd6b0a00700 1 -- 172.21.15.12:0/907857447 wait complete. 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.719+0000 7fd6b0a00700 1 Processor -- start 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.719+0000 7fd6b0a00700 1 -- start start 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.719+0000 7fd6b0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6ac104500 0x7fd6ac1a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.719+0000 7fd6b0a00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6ac104ec0 0x7fd6ac1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.719+0000 7fd6b0a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6ac10a5b0 0x7fd6ac19aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.719+0000 7fd6b0a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac114840 con 0x7fd6ac104500 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.719+0000 7fd6b0a00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac1146c0 con 0x7fd6ac104ec0 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.719+0000 7fd6b0a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac1149c0 con 0x7fd6ac10a5b0 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.720+0000 7fd6a9400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6ac104ec0 0x7fd6ac1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.720+0000 7fd6a9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6ac104500 0x7fd6ac1a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.720+0000 7fd6a9400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6ac104ec0 0x7fd6ac1a0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:55008/0 (socket says 172.21.15.12:55008) 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.720+0000 7fd6a9400700 1 -- 172.21.15.12:0/2082132910 learned_addr learned my addr 172.21.15.12:0/2082132910 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.720+0000 7fd6aa800700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6ac10a5b0 0x7fd6ac19aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.720+0000 7fd6a9400700 1 -- 172.21.15.12:0/2082132910 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6ac10a5b0 msgr2=0x7fd6ac19aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.720+0000 7fd6a9400700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6ac10a5b0 0x7fd6ac19aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.720+0000 7fd6a9400700 1 -- 172.21.15.12:0/2082132910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6ac104500 msgr2=0x7fd6ac1a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.720+0000 7fd6a9400700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6ac104500 0x7fd6ac1a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.720+0000 7fd6a9400700 1 -- 172.21.15.12:0/2082132910 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ac19b180 con 0x7fd6ac104ec0 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.720+0000 7fd6aa800700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6ac10a5b0 0x7fd6ac19aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.720+0000 7fd6a9400700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6ac104ec0 0x7fd6ac1a0da0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd698005910 tx=0x7fd69800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.721+0000 7fd6a2a00700 1 -- 172.21.15.12:0/2082132910 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd698027070 con 0x7fd6ac104ec0 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.721+0000 7fd6a2a00700 1 -- 172.21.15.12:0/2082132910 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd698003680 con 0x7fd6ac104ec0 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.721+0000 7fd6b0a00700 1 -- 172.21.15.12:0/2082132910 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6ac19b410 con 0x7fd6ac104ec0 2024-03-17T23:56:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.721+0000 7fd6a2a00700 1 -- 172.21.15.12:0/2082132910 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6980173f0 con 0x7fd6ac104ec0 2024-03-17T23:56:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.721+0000 7fd6b0a00700 1 -- 172.21.15.12:0/2082132910 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6ac1a7730 con 0x7fd6ac104ec0 2024-03-17T23:56:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.722+0000 7fd6b0a00700 1 -- 172.21.15.12:0/2082132910 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd6ac062a60 con 0x7fd6ac104ec0 2024-03-17T23:56:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.724+0000 7fd6a2a00700 1 -- 172.21.15.12:0/2082132910 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd698022470 con 0x7fd6ac104ec0 2024-03-17T23:56:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.725+0000 7fd6a2a00700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd6900a2920 0x7fd6900a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.725+0000 7fd6a2a00700 1 -- 172.21.15.12:0/2082132910 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fd6980cfed0 con 0x7fd6ac104ec0 2024-03-17T23:56:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.725+0000 7fd6a9e00700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd6900a2920 0x7fd6900a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.725+0000 7fd6a2a00700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856] conn(0x7fd6900ada80 0x7fd6900aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.725+0000 7fd6a2a00700 1 -- 172.21.15.12:0/2082132910 --> [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd69800b790 con 0x7fd6900ada80 2024-03-17T23:56:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.726+0000 7fd6a2a00700 1 -- 172.21.15.12:0/2082132910 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fd6980d0280 con 0x7fd6ac104ec0 2024-03-17T23:56:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.726+0000 7fd6aa800700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856] conn(0x7fd6900ada80 0x7fd6900aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.726+0000 7fd6a9e00700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd6900a2920 0x7fd6900a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd69c005950 tx=0x7fd69c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.726+0000 7fd6aa800700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856] conn(0x7fd6900ada80 0x7fd6900aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.727+0000 7fd6a2a00700 1 -- 172.21.15.12:0/2082132910 <== osd.7 v2:172.21.15.79:6816/1956068856 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd69800b790 con 0x7fd6900ada80 2024-03-17T23:56:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.745+0000 7fd6b0a00700 1 -- 172.21.15.12:0/2082132910 --> [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd6ac062ee0 con 0x7fd6900ada80 2024-03-17T23:56:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.746+0000 7fd6a2a00700 1 -- 172.21.15.12:0/2082132910 <== osd.7 v2:172.21.15.79:6816/1956068856 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd6ac062ee0 con 0x7fd6900ada80 2024-03-17T23:56:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.747+0000 7fd6b0a00700 1 -- 172.21.15.12:0/2082132910 >> [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856] conn(0x7fd6900ada80 msgr2=0x7fd6900aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.747+0000 7fd6b0a00700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856] conn(0x7fd6900ada80 0x7fd6900aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.747+0000 7fd6b0a00700 1 -- 172.21.15.12:0/2082132910 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd6900a2920 msgr2=0x7fd6900a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.747+0000 7fd6b0a00700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd6900a2920 0x7fd6900a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd69c005950 tx=0x7fd69c00a300 comp rx=0 tx=0).stop 2024-03-17T23:56:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.747+0000 7fd6b0a00700 1 -- 172.21.15.12:0/2082132910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6ac104ec0 msgr2=0x7fd6ac1a0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.747+0000 7fd6b0a00700 1 --2- 172.21.15.12:0/2082132910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6ac104ec0 0x7fd6ac1a0da0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd698005910 tx=0x7fd69800fa00 comp rx=0 tx=0).stop 2024-03-17T23:56:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.747+0000 7fd6aa800700 1 -- 172.21.15.12:0/2082132910 reap_dead start 2024-03-17T23:56:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.747+0000 7fd6b0a00700 1 -- 172.21.15.12:0/2082132910 shutdown_connections 2024-03-17T23:56:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.747+0000 7fd6b0a00700 1 -- 172.21.15.12:0/2082132910 >> 172.21.15.12:0/2082132910 conn(0x7fd6ac0fffd0 msgr2=0x7fd6ac101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.747+0000 7fd6b0a00700 1 -- 172.21.15.12:0/2082132910 shutdown_connections 2024-03-17T23:56:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:07.747+0000 7fd6b0a00700 1 -- 172.21.15.12:0/2082132910 wait complete. 2024-03-17T23:56:11.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:11 smithi191 ceph-mon[36708]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 9 op/s; 24 B/s, 0 keys/s, 1 objects/s recovering 2024-03-17T23:56:11.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:11 smithi012 ceph-mon[31541]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 9 op/s; 24 B/s, 0 keys/s, 1 objects/s recovering 2024-03-17T23:56:12.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:11 smithi079 ceph-mon[36241]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 9 op/s; 24 B/s, 0 keys/s, 1 objects/s recovering 2024-03-17T23:56:12.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.276+0000 7f80ddc00700 1 -- 172.21.15.12:0/2392052639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80d8108320 msgr2=0x7f80d8108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:12.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.276+0000 7f80ddc00700 1 --2- 172.21.15.12:0/2392052639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80d8108320 0x7f80d8108700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009990 tx=0x7f80c8009ca0 comp rx=0 tx=0).stop 2024-03-17T23:56:12.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.276+0000 7f80ddc00700 1 -- 172.21.15.12:0/2392052639 shutdown_connections 2024-03-17T23:56:12.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.276+0000 7f80ddc00700 1 --2- 172.21.15.12:0/2392052639 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80d806a920 0x7f80d810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.276+0000 7f80ddc00700 1 --2- 172.21.15.12:0/2392052639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80d8108320 0x7f80d8108700 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.276+0000 7f80ddc00700 1 --2- 172.21.15.12:0/2392052639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80d8069f80 0x7f80d806a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.276+0000 7f80ddc00700 1 -- 172.21.15.12:0/2392052639 >> 172.21.15.12:0/2392052639 conn(0x7f80d8078100 msgr2=0x7f80d8078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.276+0000 7f80ddc00700 1 -- 172.21.15.12:0/2392052639 shutdown_connections 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.278+0000 7f80ddc00700 1 -- 172.21.15.12:0/2392052639 wait complete. 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.279+0000 7f80ddc00700 1 Processor -- start 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.279+0000 7f80ddc00700 1 -- start start 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.279+0000 7f80ddc00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80d8069f80 0x7f80d819e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.279+0000 7f80ddc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80d806a920 0x7f80d819ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80ddc00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80d8108320 0x7f80d8198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80ddc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f80d8112630 con 0x7f80d806a920 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80ddc00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f80d81124b0 con 0x7f80d8069f80 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80ddc00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f80d81127b0 con 0x7f80d8108320 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80d6a00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80d8069f80 0x7f80d819e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:12.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80d6a00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80d8069f80 0x7f80d819e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:55040/0 (socket says 172.21.15.12:55040) 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80d6a00700 1 -- 172.21.15.12:0/3671847541 learned_addr learned my addr 172.21.15.12:0/3671847541 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80d6000700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80d806a920 0x7f80d819ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80d7400700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80d8108320 0x7f80d8198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80d6a00700 1 -- 172.21.15.12:0/3671847541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80d8108320 msgr2=0x7f80d8198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80d6a00700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80d8108320 0x7f80d8198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80d6a00700 1 -- 172.21.15.12:0/3671847541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80d806a920 msgr2=0x7f80d819ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80d6a00700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80d806a920 0x7f80d819ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.280+0000 7f80d6a00700 1 -- 172.21.15.12:0/3671847541 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d8199090 con 0x7f80d8069f80 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.281+0000 7f80d6a00700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80d8069f80 0x7f80d819e770 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f80c8000c00 tx=0x7f80c8003890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.281+0000 7f80cf400700 1 -- 172.21.15.12:0/3671847541 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c8027070 con 0x7f80d8069f80 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.281+0000 7f80cf400700 1 -- 172.21.15.12:0/3671847541 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80c8003e40 con 0x7f80d8069f80 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.281+0000 7f80cf400700 1 -- 172.21.15.12:0/3671847541 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c8022c90 con 0x7f80d8069f80 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.281+0000 7f80ddc00700 1 -- 172.21.15.12:0/3671847541 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80d8199320 con 0x7f80d8069f80 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.281+0000 7f80ddc00700 1 -- 172.21.15.12:0/3671847541 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80d81a5640 con 0x7f80d8069f80 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.282+0000 7f80ddc00700 1 -- 172.21.15.12:0/3671847541 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f80d8062a60 con 0x7f80d8069f80 2024-03-17T23:56:12.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.285+0000 7f80cf400700 1 -- 172.21.15.12:0/3671847541 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f80c8003ff0 con 0x7f80d8069f80 2024-03-17T23:56:12.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.286+0000 7f80cf400700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f80bc0a29a0 0x7f80bc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:12.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.287+0000 7f80cf400700 1 -- 172.21.15.12:0/3671847541 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f80c80d0d80 con 0x7f80d8069f80 2024-03-17T23:56:12.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.287+0000 7f80d6000700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f80bc0a29a0 0x7f80bc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:12.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.287+0000 7f80cf400700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796] conn(0x7f80bc0adb00 0x7f80bc0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:12.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.287+0000 7f80cf400700 1 -- 172.21.15.12:0/3671847541 --> [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80c800dec0 con 0x7f80bc0adb00 2024-03-17T23:56:12.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.287+0000 7f80cf400700 1 -- 172.21.15.12:0/3671847541 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f80c80d1130 con 0x7f80d8069f80 2024-03-17T23:56:12.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.288+0000 7f80d7400700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796] conn(0x7f80bc0adb00 0x7f80bc0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:12.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.288+0000 7f80d6000700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f80bc0a29a0 0x7f80bc0a4e60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f80d8078770 tx=0x7f80c4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:12.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.288+0000 7f80d7400700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796] conn(0x7f80bc0adb00 0x7f80bc0affa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:12.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.289+0000 7f80cf400700 1 -- 172.21.15.12:0/3671847541 <== osd.9 v2:172.21.15.12:6826/140672796 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80c800dec0 con 0x7f80bc0adb00 2024-03-17T23:56:12.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.308+0000 7f80ddc00700 1 -- 172.21.15.12:0/3671847541 --> [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f80d8062ee0 con 0x7f80bc0adb00 2024-03-17T23:56:12.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.309+0000 7f80cf400700 1 -- 172.21.15.12:0/3671847541 <== osd.9 v2:172.21.15.12:6826/140672796 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f80d8062ee0 con 0x7f80bc0adb00 2024-03-17T23:56:12.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.309+0000 7f80ddc00700 1 -- 172.21.15.12:0/3671847541 >> [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796] conn(0x7f80bc0adb00 msgr2=0x7f80bc0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:12.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.309+0000 7f80ddc00700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796] conn(0x7f80bc0adb00 0x7f80bc0affa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:12.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.309+0000 7f80ddc00700 1 -- 172.21.15.12:0/3671847541 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f80bc0a29a0 msgr2=0x7f80bc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:12.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.309+0000 7f80ddc00700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f80bc0a29a0 0x7f80bc0a4e60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f80d8078770 tx=0x7f80c4009190 comp rx=0 tx=0).stop 2024-03-17T23:56:12.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.309+0000 7f80ddc00700 1 -- 172.21.15.12:0/3671847541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80d8069f80 msgr2=0x7f80d819e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:12.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.309+0000 7f80ddc00700 1 --2- 172.21.15.12:0/3671847541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80d8069f80 0x7f80d819e770 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f80c8000c00 tx=0x7f80c8003890 comp rx=0 tx=0).stop 2024-03-17T23:56:12.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.309+0000 7f80d7400700 1 -- 172.21.15.12:0/3671847541 reap_dead start 2024-03-17T23:56:12.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.310+0000 7f80ddc00700 1 -- 172.21.15.12:0/3671847541 shutdown_connections 2024-03-17T23:56:12.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.310+0000 7f80ddc00700 1 -- 172.21.15.12:0/3671847541 >> 172.21.15.12:0/3671847541 conn(0x7f80d8078100 msgr2=0x7f80d80ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:12.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.310+0000 7f80ddc00700 1 -- 172.21.15.12:0/3671847541 shutdown_connections 2024-03-17T23:56:12.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:10.310+0000 7f80ddc00700 1 -- 172.21.15.12:0/3671847541 wait complete. 2024-03-17T23:56:13.277 INFO:teuthology.orchestra.run.smithi012.stdout:219043332146 2024-03-17T23:56:13.278 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-17T23:56:13.760 INFO:teuthology.orchestra.run.smithi012.stdout:77309411407 2024-03-17T23:56:13.761 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-17T23:56:13.823 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:13 smithi012 ceph-mon[31541]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-17T23:56:13.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:13 smithi191 ceph-mon[36708]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-17T23:56:14.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:13 smithi079 ceph-mon[36241]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-17T23:56:14.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.533+0000 7f6d43e00700 1 -- 172.21.15.12:0/782182848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d3c10a570 msgr2=0x7f6d3c10a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:14.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.533+0000 7f6d43e00700 1 --2- 172.21.15.12:0/782182848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d3c10a570 0x7f6d3c10a950 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009990 tx=0x7f6d30009ca0 comp rx=0 tx=0).stop 2024-03-17T23:56:14.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.533+0000 7f6d43e00700 1 -- 172.21.15.12:0/782182848 shutdown_connections 2024-03-17T23:56:14.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.533+0000 7f6d43e00700 1 --2- 172.21.15.12:0/782182848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d3c104e80 0x7f6d3c111340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:14.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.533+0000 7f6d43e00700 1 --2- 172.21.15.12:0/782182848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d3c10a570 0x7f6d3c10a950 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:14.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.533+0000 7f6d43e00700 1 --2- 172.21.15.12:0/782182848 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d3c1044c0 0x7f6d3c104940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:14.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.533+0000 7f6d43e00700 1 -- 172.21.15.12:0/782182848 >> 172.21.15.12:0/782182848 conn(0x7f6d3c0fffb0 msgr2=0x7f6d3c1023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:14.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.534+0000 7f6d43e00700 1 -- 172.21.15.12:0/782182848 shutdown_connections 2024-03-17T23:56:14.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.534+0000 7f6d43e00700 1 -- 172.21.15.12:0/782182848 wait complete. 2024-03-17T23:56:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.535+0000 7f6d43e00700 1 Processor -- start 2024-03-17T23:56:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.535+0000 7f6d43e00700 1 -- start start 2024-03-17T23:56:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.536+0000 7f6d43e00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d3c1044c0 0x7f6d3c1a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.536+0000 7f6d43e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d3c104e80 0x7f6d3c1a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.536+0000 7f6d43e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d3c10a570 0x7f6d3c19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.536+0000 7f6d43e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3c114820 con 0x7f6d3c10a570 2024-03-17T23:56:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.536+0000 7f6d43e00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3c1146a0 con 0x7f6d3c1044c0 2024-03-17T23:56:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.536+0000 7f6d43e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3c1149a0 con 0x7f6d3c104e80 2024-03-17T23:56:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.536+0000 7f6d41400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d3c1044c0 0x7f6d3c1a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.536+0000 7f6d40a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d3c104e80 0x7f6d3c1a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.536+0000 7f6d41400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d3c1044c0 0x7f6d3c1a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:33284/0 (socket says 172.21.15.12:33284) 2024-03-17T23:56:14.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.536+0000 7f6d41400700 1 -- 172.21.15.12:0/1615917663 learned_addr learned my addr 172.21.15.12:0/1615917663 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:14.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.536+0000 7f6d41e00700 1 --2- 172.21.15.12:0/1615917663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d3c10a570 0x7f6d3c19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:14.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.537+0000 7f6d41400700 1 -- 172.21.15.12:0/1615917663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d3c104e80 msgr2=0x7f6d3c1a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:14.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.537+0000 7f6d41400700 1 --2- 172.21.15.12:0/1615917663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d3c104e80 0x7f6d3c1a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:14.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.537+0000 7f6d41400700 1 -- 172.21.15.12:0/1615917663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d3c10a570 msgr2=0x7f6d3c19aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:14.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.537+0000 7f6d41400700 1 --2- 172.21.15.12:0/1615917663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6d3c10a570 0x7f6d3c19aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:14.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.537+0000 7f6d41400700 1 -- 172.21.15.12:0/1615917663 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d3c19b1c0 con 0x7f6d3c1044c0 2024-03-17T23:56:14.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.537+0000 7f6d41400700 1 --2- 172.21.15.12:0/1615917663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d3c1044c0 0x7f6d3c1a08a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6d30005910 tx=0x7f6d300047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:14.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.537+0000 7f6d3a000700 1 -- 172.21.15.12:0/1615917663 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d30027070 con 0x7f6d3c1044c0 2024-03-17T23:56:14.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.537+0000 7f6d3a000700 1 -- 172.21.15.12:0/1615917663 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d3000b6c0 con 0x7f6d3c1044c0 2024-03-17T23:56:14.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.537+0000 7f6d3a000700 1 -- 172.21.15.12:0/1615917663 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d30017650 con 0x7f6d3c1044c0 2024-03-17T23:56:14.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.537+0000 7f6d43e00700 1 -- 172.21.15.12:0/1615917663 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d3c19b450 con 0x7f6d3c1044c0 2024-03-17T23:56:14.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.537+0000 7f6d43e00700 1 -- 172.21.15.12:0/1615917663 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d3c1a7770 con 0x7f6d3c1044c0 2024-03-17T23:56:14.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.538+0000 7f6d43e00700 1 -- 172.21.15.12:0/1615917663 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6d3c062a20 con 0x7f6d3c1044c0 2024-03-17T23:56:14.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.540+0000 7f6d3a000700 1 -- 172.21.15.12:0/1615917663 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6d3002a070 con 0x7f6d3c1044c0 2024-03-17T23:56:14.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.543+0000 7f6d3a000700 1 --2- 172.21.15.12:0/1615917663 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6d240a29a0 0x7f6d240a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:14.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.543+0000 7f6d3a000700 1 -- 172.21.15.12:0/1615917663 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f6d300cff00 con 0x7f6d3c1044c0 2024-03-17T23:56:14.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.543+0000 7f6d40a00700 1 --2- 172.21.15.12:0/1615917663 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6d240a29a0 0x7f6d240a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:14.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.544+0000 7f6d3a000700 1 --2- 172.21.15.12:0/1615917663 >> [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194] conn(0x7f6d240adb00 0x7f6d240affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:14.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.544+0000 7f6d3a000700 1 -- 172.21.15.12:0/1615917663 --> [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d3000fe50 con 0x7f6d240adb00 2024-03-17T23:56:14.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.544+0000 7f6d3a000700 1 -- 172.21.15.12:0/1615917663 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f6d300d2520 con 0x7f6d3c1044c0 2024-03-17T23:56:14.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.544+0000 7f6d41e00700 1 --2- 172.21.15.12:0/1615917663 >> [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194] conn(0x7f6d240adb00 0x7f6d240affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:14.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.544+0000 7f6d40a00700 1 --2- 172.21.15.12:0/1615917663 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6d240a29a0 0x7f6d240a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c109af0 tx=0x7f6d28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:14.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.544+0000 7f6d41e00700 1 --2- 172.21.15.12:0/1615917663 >> [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194] conn(0x7f6d240adb00 0x7f6d240affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:14.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.545+0000 7f6d3a000700 1 -- 172.21.15.12:0/1615917663 <== osd.5 v2:172.21.15.191:6808/862706194 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d3000fe50 con 0x7f6d240adb00 2024-03-17T23:56:14.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.568+0000 7f6d43e00700 1 -- 172.21.15.12:0/1615917663 --> [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6d3c062f20 con 0x7f6d240adb00 2024-03-17T23:56:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.568+0000 7f6d3a000700 1 -- 172.21.15.12:0/1615917663 <== osd.5 v2:172.21.15.191:6808/862706194 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6d3c062f20 con 0x7f6d240adb00 2024-03-17T23:56:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.569+0000 7f6d43e00700 1 -- 172.21.15.12:0/1615917663 >> [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194] conn(0x7f6d240adb00 msgr2=0x7f6d240affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:14.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.569+0000 7f6d43e00700 1 --2- 172.21.15.12:0/1615917663 >> [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194] conn(0x7f6d240adb00 0x7f6d240affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:14.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.569+0000 7f6d43e00700 1 -- 172.21.15.12:0/1615917663 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6d240a29a0 msgr2=0x7f6d240a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:14.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.569+0000 7f6d43e00700 1 --2- 172.21.15.12:0/1615917663 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6d240a29a0 0x7f6d240a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c109af0 tx=0x7f6d28009260 comp rx=0 tx=0).stop 2024-03-17T23:56:14.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.569+0000 7f6d43e00700 1 -- 172.21.15.12:0/1615917663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d3c1044c0 msgr2=0x7f6d3c1a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:14.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.569+0000 7f6d43e00700 1 --2- 172.21.15.12:0/1615917663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d3c1044c0 0x7f6d3c1a08a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6d30005910 tx=0x7f6d300047b0 comp rx=0 tx=0).stop 2024-03-17T23:56:14.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.569+0000 7f6d41e00700 1 -- 172.21.15.12:0/1615917663 reap_dead start 2024-03-17T23:56:14.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.569+0000 7f6d43e00700 1 -- 172.21.15.12:0/1615917663 shutdown_connections 2024-03-17T23:56:14.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.569+0000 7f6d43e00700 1 -- 172.21.15.12:0/1615917663 >> 172.21.15.12:0/1615917663 conn(0x7f6d3c0fffb0 msgr2=0x7f6d3c101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:14.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.570+0000 7f6d43e00700 1 -- 172.21.15.12:0/1615917663 shutdown_connections 2024-03-17T23:56:14.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:14.570+0000 7f6d43e00700 1 -- 172.21.15.12:0/1615917663 wait complete. 2024-03-17T23:56:14.853 INFO:teuthology.orchestra.run.smithi012.stdout:107374182473 2024-03-17T23:56:14.853 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-17T23:56:15.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:15 smithi012 ceph-mon[31541]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-17T23:56:15.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:15 smithi191 ceph-mon[36708]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-17T23:56:15.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.887+0000 7f951dc00700 1 -- 172.21.15.12:0/2562401324 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9518107650 msgr2=0x7f951810f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:15.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.887+0000 7f951dc00700 1 --2- 172.21.15.12:0/2562401324 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9518107650 0x7f951810f220 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f951000b0a0 tx=0x7f951000b3b0 comp rx=0 tx=0).stop 2024-03-17T23:56:15.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.887+0000 7f951dc00700 1 -- 172.21.15.12:0/2562401324 shutdown_connections 2024-03-17T23:56:15.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.887+0000 7f951dc00700 1 --2- 172.21.15.12:0/2562401324 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9518107650 0x7f951810f220 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:15.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.887+0000 7f951dc00700 1 --2- 172.21.15.12:0/2562401324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9518102580 0x7f9518102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:15.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.887+0000 7f951dc00700 1 --2- 172.21.15.12:0/2562401324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9518102f30 0x7f9518106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:15.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.887+0000 7f951dc00700 1 -- 172.21.15.12:0/2562401324 >> 172.21.15.12:0/2562401324 conn(0x7f9518079430 msgr2=0x7f9518079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:15.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.887+0000 7f951dc00700 1 -- 172.21.15.12:0/2562401324 shutdown_connections 2024-03-17T23:56:15.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.887+0000 7f951dc00700 1 -- 172.21.15.12:0/2562401324 wait complete. 2024-03-17T23:56:15.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.888+0000 7f951dc00700 1 Processor -- start 2024-03-17T23:56:15.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.889+0000 7f951dc00700 1 -- start start 2024-03-17T23:56:15.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.889+0000 7f951dc00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9518102580 0x7f951819e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.889+0000 7f951dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9518102f30 0x7f951819ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.889+0000 7f951dc00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9518107650 0x7f9518198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.889+0000 7f951dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f95181126c0 con 0x7f9518102f30 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.889+0000 7f951dc00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9518112540 con 0x7f9518107650 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.889+0000 7f951dc00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9518112840 con 0x7f9518102580 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.889+0000 7f9516a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9518102580 0x7f951819e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.889+0000 7f9516000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9518102f30 0x7f951819ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.889+0000 7f9516a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9518102580 0x7f951819e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:42034/0 (socket says 172.21.15.12:42034) 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.890+0000 7f9516a00700 1 -- 172.21.15.12:0/3227214989 learned_addr learned my addr 172.21.15.12:0/3227214989 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.890+0000 7f9517400700 1 --2- 172.21.15.12:0/3227214989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9518107650 0x7f9518198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.890+0000 7f9517400700 1 -- 172.21.15.12:0/3227214989 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9518102580 msgr2=0x7f951819e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.890+0000 7f9517400700 1 --2- 172.21.15.12:0/3227214989 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9518102580 0x7f951819e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.890+0000 7f9517400700 1 -- 172.21.15.12:0/3227214989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9518102f30 msgr2=0x7f951819ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.890+0000 7f9517400700 1 --2- 172.21.15.12:0/3227214989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9518102f30 0x7f951819ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:15.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.890+0000 7f9517400700 1 -- 172.21.15.12:0/3227214989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95181990e0 con 0x7f9518107650 2024-03-17T23:56:15.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.890+0000 7f9517400700 1 --2- 172.21.15.12:0/3227214989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9518107650 0x7f9518198980 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f95100062a0 tx=0x7f9510004c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:15.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.891+0000 7f950f400700 1 -- 172.21.15.12:0/3227214989 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9510021070 con 0x7f9518107650 2024-03-17T23:56:15.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.891+0000 7f950f400700 1 -- 172.21.15.12:0/3227214989 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9510003c00 con 0x7f9518107650 2024-03-17T23:56:15.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.891+0000 7f950f400700 1 -- 172.21.15.12:0/3227214989 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95100045a0 con 0x7f9518107650 2024-03-17T23:56:15.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.891+0000 7f951dc00700 1 -- 172.21.15.12:0/3227214989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9518199370 con 0x7f9518107650 2024-03-17T23:56:15.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.891+0000 7f951dc00700 1 -- 172.21.15.12:0/3227214989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95181a5640 con 0x7f9518107650 2024-03-17T23:56:15.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.892+0000 7f951dc00700 1 -- 172.21.15.12:0/3227214989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f95180629a0 con 0x7f9518107650 2024-03-17T23:56:15.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.896+0000 7f950f400700 1 -- 172.21.15.12:0/3227214989 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9510004740 con 0x7f9518107650 2024-03-17T23:56:15.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.897+0000 7f950f400700 1 --2- 172.21.15.12:0/3227214989 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f94fc0a29a0 0x7f94fc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:15.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.897+0000 7f950f400700 1 -- 172.21.15.12:0/3227214989 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f95100cfff0 con 0x7f9518107650 2024-03-17T23:56:15.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.897+0000 7f9516a00700 1 --2- 172.21.15.12:0/3227214989 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f94fc0a29a0 0x7f94fc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:15.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.897+0000 7f950f400700 1 --2- 172.21.15.12:0/3227214989 >> [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303] conn(0x7f94fc0adb00 0x7f94fc0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:15.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.898+0000 7f950f400700 1 -- 172.21.15.12:0/3227214989 --> [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9510005a50 con 0x7f94fc0adb00 2024-03-17T23:56:15.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.898+0000 7f950f400700 1 -- 172.21.15.12:0/3227214989 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f95100d0400 con 0x7f9518107650 2024-03-17T23:56:15.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.898+0000 7f9516000700 1 --2- 172.21.15.12:0/3227214989 >> [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303] conn(0x7f94fc0adb00 0x7f94fc0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:15.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.898+0000 7f9516a00700 1 --2- 172.21.15.12:0/3227214989 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f94fc0a29a0 0x7f94fc0a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9518079bc0 tx=0x7f9508007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:15.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.898+0000 7f9516000700 1 --2- 172.21.15.12:0/3227214989 >> [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303] conn(0x7f94fc0adb00 0x7f94fc0affa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:15.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.899+0000 7f950f400700 1 -- 172.21.15.12:0/3227214989 <== osd.11 v2:172.21.15.191:6824/1030815303 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9510005a50 con 0x7f94fc0adb00 2024-03-17T23:56:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.929+0000 7f951dc00700 1 -- 172.21.15.12:0/3227214989 --> [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9518062e20 con 0x7f94fc0adb00 2024-03-17T23:56:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.929+0000 7f950f400700 1 -- 172.21.15.12:0/3227214989 <== osd.11 v2:172.21.15.191:6824/1030815303 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9518062e20 con 0x7f94fc0adb00 2024-03-17T23:56:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.930+0000 7f951dc00700 1 -- 172.21.15.12:0/3227214989 >> [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303] conn(0x7f94fc0adb00 msgr2=0x7f94fc0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.930+0000 7f951dc00700 1 --2- 172.21.15.12:0/3227214989 >> [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303] conn(0x7f94fc0adb00 0x7f94fc0affa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.930+0000 7f951dc00700 1 -- 172.21.15.12:0/3227214989 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f94fc0a29a0 msgr2=0x7f94fc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.930+0000 7f951dc00700 1 --2- 172.21.15.12:0/3227214989 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f94fc0a29a0 0x7f94fc0a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9518079bc0 tx=0x7f9508007480 comp rx=0 tx=0).stop 2024-03-17T23:56:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.930+0000 7f951dc00700 1 -- 172.21.15.12:0/3227214989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9518107650 msgr2=0x7f9518198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.930+0000 7f951dc00700 1 --2- 172.21.15.12:0/3227214989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9518107650 0x7f9518198980 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f95100062a0 tx=0x7f9510004c60 comp rx=0 tx=0).stop 2024-03-17T23:56:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.930+0000 7f9517400700 1 -- 172.21.15.12:0/3227214989 reap_dead start 2024-03-17T23:56:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.930+0000 7f951dc00700 1 -- 172.21.15.12:0/3227214989 shutdown_connections 2024-03-17T23:56:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.930+0000 7f951dc00700 1 -- 172.21.15.12:0/3227214989 >> 172.21.15.12:0/3227214989 conn(0x7f9518079430 msgr2=0x7f9518100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.931+0000 7f951dc00700 1 -- 172.21.15.12:0/3227214989 shutdown_connections 2024-03-17T23:56:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:15.931+0000 7f951dc00700 1 -- 172.21.15.12:0/3227214989 wait complete. 2024-03-17T23:56:16.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:15 smithi079 ceph-mon[36241]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-17T23:56:16.415 INFO:teuthology.orchestra.run.smithi012.stdout:287762808872 2024-03-17T23:56:16.415 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-17T23:56:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.142+0000 7fa4c5800700 1 -- 172.21.15.12:0/3985944382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4c0108320 msgr2=0x7fa4c0108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.142+0000 7fa4c5800700 1 --2- 172.21.15.12:0/3985944382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4c0108320 0x7fa4c0108700 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa4b000b0d0 tx=0x7fa4b000b3e0 comp rx=0 tx=0).stop 2024-03-17T23:56:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.143+0000 7fa4c5800700 1 -- 172.21.15.12:0/3985944382 shutdown_connections 2024-03-17T23:56:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.143+0000 7fa4c5800700 1 --2- 172.21.15.12:0/3985944382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c006a920 0x7fa4c010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.143+0000 7fa4c5800700 1 --2- 172.21.15.12:0/3985944382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4c0108320 0x7fa4c0108700 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.143+0000 7fa4c5800700 1 --2- 172.21.15.12:0/3985944382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c0069f80 0x7fa4c006a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.143+0000 7fa4c5800700 1 -- 172.21.15.12:0/3985944382 >> 172.21.15.12:0/3985944382 conn(0x7fa4c0078100 msgr2=0x7fa4c0078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.143+0000 7fa4c5800700 1 -- 172.21.15.12:0/3985944382 shutdown_connections 2024-03-17T23:56:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.143+0000 7fa4c5800700 1 -- 172.21.15.12:0/3985944382 wait complete. 2024-03-17T23:56:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.144+0000 7fa4c5800700 1 Processor -- start 2024-03-17T23:56:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.144+0000 7fa4c5800700 1 -- start start 2024-03-17T23:56:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.145+0000 7fa4c5800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c0069f80 0x7fa4c01012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.145+0000 7fa4c5800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4c006a920 0x7fa4c01017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.145+0000 7fa4c5800700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c0108320 0x7fa4c0101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.145+0000 7fa4c5800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c0112580 con 0x7fa4c0069f80 2024-03-17T23:56:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.145+0000 7fa4c5800700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c0112400 con 0x7fa4c0108320 2024-03-17T23:56:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.145+0000 7fa4c5800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c0112700 con 0x7fa4c006a920 2024-03-17T23:56:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.145+0000 7fa4bf400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c0108320 0x7fa4c0101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.145+0000 7fa4bf400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c0108320 0x7fa4c0101d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:33326/0 (socket says 172.21.15.12:33326) 2024-03-17T23:56:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.145+0000 7fa4bf400700 1 -- 172.21.15.12:0/3652728086 learned_addr learned my addr 172.21.15.12:0/3652728086 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.145+0000 7fa4bea00700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c0069f80 0x7fa4c01012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.145+0000 7fa4b6000700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4c006a920 0x7fa4c01017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.146+0000 7fa4bf400700 1 -- 172.21.15.12:0/3652728086 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4c006a920 msgr2=0x7fa4c01017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.146+0000 7fa4bf400700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4c006a920 0x7fa4c01017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.146+0000 7fa4bf400700 1 -- 172.21.15.12:0/3652728086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c0069f80 msgr2=0x7fa4c01012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.146+0000 7fa4bf400700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4c0069f80 0x7fa4c01012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.146+0000 7fa4bf400700 1 -- 172.21.15.12:0/3652728086 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c01ab420 con 0x7fa4c0108320 2024-03-17T23:56:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.146+0000 7fa4bf400700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c0108320 0x7fa4c0101d20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800e940 tx=0x7fa4b800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.146+0000 7fa4b7e00700 1 -- 172.21.15.12:0/3652728086 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8004a30 con 0x7fa4c0108320 2024-03-17T23:56:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.146+0000 7fa4b7e00700 1 -- 172.21.15.12:0/3652728086 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4b8005500 con 0x7fa4c0108320 2024-03-17T23:56:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.146+0000 7fa4b7e00700 1 -- 172.21.15.12:0/3652728086 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b800ce20 con 0x7fa4c0108320 2024-03-17T23:56:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.146+0000 7fa4c5800700 1 -- 172.21.15.12:0/3652728086 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c01ab6b0 con 0x7fa4c0108320 2024-03-17T23:56:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.146+0000 7fa4c5800700 1 -- 172.21.15.12:0/3652728086 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c01abba0 con 0x7fa4c0108320 2024-03-17T23:56:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.147+0000 7fa4c5800700 1 -- 172.21.15.12:0/3652728086 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa4c00629e0 con 0x7fa4c0108320 2024-03-17T23:56:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.151+0000 7fa4b7e00700 1 -- 172.21.15.12:0/3652728086 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa4b8005020 con 0x7fa4c0108320 2024-03-17T23:56:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.152+0000 7fa4b7e00700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4a40a29a0 0x7fa4a40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.153+0000 7fa4b7e00700 1 -- 172.21.15.12:0/3652728086 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fa4b80cf0a0 con 0x7fa4c0108320 2024-03-17T23:56:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.153+0000 7fa4bea00700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4a40a29a0 0x7fa4a40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.153+0000 7fa4b7e00700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279] conn(0x7fa4a40adb00 0x7fa4a40affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.153+0000 7fa4b7e00700 1 -- 172.21.15.12:0/3652728086 --> [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa4a40b0670 con 0x7fa4a40adb00 2024-03-17T23:56:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.153+0000 7fa4b7e00700 1 -- 172.21.15.12:0/3652728086 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fa4b80d6050 con 0x7fa4c0108320 2024-03-17T23:56:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.153+0000 7fa4b6000700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279] conn(0x7fa4a40adb00 0x7fa4a40affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.154+0000 7fa4bea00700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4a40a29a0 0x7fa4a40a4e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0009fd0 tx=0x7fa4b000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.154+0000 7fa4b6000700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279] conn(0x7fa4a40adb00 0x7fa4a40affa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.156+0000 7fa4b7e00700 1 -- 172.21.15.12:0/3652728086 <== osd.0 v2:172.21.15.12:6802/3127075279 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa4a40b0670 con 0x7fa4a40adb00 2024-03-17T23:56:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.174+0000 7fa4c5800700 1 -- 172.21.15.12:0/3652728086 --> [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa4c0062de0 con 0x7fa4a40adb00 2024-03-17T23:56:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.174+0000 7fa4b7e00700 1 -- 172.21.15.12:0/3652728086 <== osd.0 v2:172.21.15.12:6802/3127075279 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa4c0062de0 con 0x7fa4a40adb00 2024-03-17T23:56:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.175+0000 7fa4c5800700 1 -- 172.21.15.12:0/3652728086 >> [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279] conn(0x7fa4a40adb00 msgr2=0x7fa4a40affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.175+0000 7fa4c5800700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279] conn(0x7fa4a40adb00 0x7fa4a40affa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.175+0000 7fa4c5800700 1 -- 172.21.15.12:0/3652728086 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4a40a29a0 msgr2=0x7fa4a40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.175+0000 7fa4c5800700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4a40a29a0 0x7fa4a40a4e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0009fd0 tx=0x7fa4b000b040 comp rx=0 tx=0).stop 2024-03-17T23:56:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.175+0000 7fa4c5800700 1 -- 172.21.15.12:0/3652728086 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c0108320 msgr2=0x7fa4c0101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.175+0000 7fa4c5800700 1 --2- 172.21.15.12:0/3652728086 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c0108320 0x7fa4c0101d20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800e940 tx=0x7fa4b800ec50 comp rx=0 tx=0).stop 2024-03-17T23:56:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.175+0000 7fa4bf400700 1 -- 172.21.15.12:0/3652728086 reap_dead start 2024-03-17T23:56:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.175+0000 7fa4c5800700 1 -- 172.21.15.12:0/3652728086 shutdown_connections 2024-03-17T23:56:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.175+0000 7fa4c5800700 1 -- 172.21.15.12:0/3652728086 >> 172.21.15.12:0/3652728086 conn(0x7fa4c0078100 msgr2=0x7fa4c00ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.176+0000 7fa4c5800700 1 -- 172.21.15.12:0/3652728086 shutdown_connections 2024-03-17T23:56:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:16.176+0000 7fa4c5800700 1 -- 172.21.15.12:0/3652728086 wait complete. 2024-03-17T23:56:17.486 INFO:teuthology.orchestra.run.smithi012.stdout:163208757312 2024-03-17T23:56:17.486 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-17T23:56:17.832 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:17 smithi012 ceph-mon[31541]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:56:17.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:17 smithi191 ceph-mon[36708]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:56:17.847 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:17 smithi079 ceph-mon[36241]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:56:18.099 INFO:teuthology.orchestra.run.smithi012.stdout:34359738459 2024-03-17T23:56:18.100 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-17T23:56:18.456 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:18.461 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:18.461 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:18.469 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:18.470 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:18.491 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:18.495 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:18.532 INFO:teuthology.orchestra.run.smithi012.stdout:343597383710 2024-03-17T23:56:18.532 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-17T23:56:18.638 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:18.688 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:19.563 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:19 smithi012 ceph-mon[31541]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:19.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:19 smithi191 ceph-mon[36708]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:20.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:19 smithi079 ceph-mon[36241]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:21.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:21 smithi012 ceph-mon[31541]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:21.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:21 smithi191 ceph-mon[36708]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.965+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2303447884 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ec8104cd0 msgr2=0x7f3ec8105160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.965+0000 7f3ecfa00700 1 --2- 172.21.15.12:0/2303447884 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ec8104cd0 0x7f3ec8105160 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000b0a0 tx=0x7f3ec000b3b0 comp rx=0 tx=0).stop 2024-03-17T23:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.965+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2303447884 shutdown_connections 2024-03-17T23:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.965+0000 7f3ecfa00700 1 --2- 172.21.15.12:0/2303447884 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ec8104cd0 0x7f3ec8105160 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.965+0000 7f3ecfa00700 1 --2- 172.21.15.12:0/2303447884 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ec81071c0 0x7f3ec8069f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.965+0000 7f3ecfa00700 1 --2- 172.21.15.12:0/2303447884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec806a4c0 0x7f3ec806a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.965+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2303447884 >> 172.21.15.12:0/2303447884 conn(0x7f3ec8078100 msgr2=0x7f3ec8078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.965+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2303447884 shutdown_connections 2024-03-17T23:56:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.965+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2303447884 wait complete. 2024-03-17T23:56:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.967+0000 7f3ecfa00700 1 Processor -- start 2024-03-17T23:56:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.967+0000 7f3ecfa00700 1 -- start start 2024-03-17T23:56:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.967+0000 7f3ecfa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec806a4c0 0x7f3ec819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.967+0000 7f3ecfa00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ec8104cd0 0x7f3ec819f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.967+0000 7f3ecfa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ec81071c0 0x7f3ec81a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.967+0000 7f3ecfa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec8118ba0 con 0x7f3ec806a4c0 2024-03-17T23:56:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.967+0000 7f3ecfa00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec8118a20 con 0x7f3ec8104cd0 2024-03-17T23:56:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.967+0000 7f3ecfa00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec8118d20 con 0x7f3ec81071c0 2024-03-17T23:56:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.968+0000 7f3ec7e00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ec8104cd0 0x7f3ec819f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.968+0000 7f3ecd000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec806a4c0 0x7f3ec819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.968+0000 7f3ec7e00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ec8104cd0 0x7f3ec819f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:33356/0 (socket says 172.21.15.12:33356) 2024-03-17T23:56:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.968+0000 7f3ec7e00700 1 -- 172.21.15.12:0/2132954429 learned_addr learned my addr 172.21.15.12:0/2132954429 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.968+0000 7f3ecda00700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ec81071c0 0x7f3ec81a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.968+0000 7f3ecda00700 1 -- 172.21.15.12:0/2132954429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ec8104cd0 msgr2=0x7f3ec819f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.968+0000 7f3ecda00700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ec8104cd0 0x7f3ec819f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.968+0000 7f3ecda00700 1 -- 172.21.15.12:0/2132954429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec806a4c0 msgr2=0x7f3ec819f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.968+0000 7f3ecda00700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec806a4c0 0x7f3ec819f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.968+0000 7f3ecda00700 1 -- 172.21.15.12:0/2132954429 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec81a3fd0 con 0x7f3ec81071c0 2024-03-17T23:56:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.968+0000 7f3ec7e00700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ec8104cd0 0x7f3ec819f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.969+0000 7f3ecda00700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ec81071c0 0x7f3ec81a3890 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000b070 tx=0x7f3ec0004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.969+0000 7f3ec5600700 1 -- 172.21.15.12:0/2132954429 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0021070 con 0x7f3ec81071c0 2024-03-17T23:56:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.969+0000 7f3ec5600700 1 -- 172.21.15.12:0/2132954429 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ec0007b80 con 0x7f3ec81071c0 2024-03-17T23:56:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.969+0000 7f3ec5600700 1 -- 172.21.15.12:0/2132954429 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0005040 con 0x7f3ec81071c0 2024-03-17T23:56:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.969+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2132954429 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec81a4260 con 0x7f3ec81071c0 2024-03-17T23:56:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.969+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2132954429 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec81abb40 con 0x7f3ec81071c0 2024-03-17T23:56:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.970+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2132954429 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ec8068f90 con 0x7f3ec81071c0 2024-03-17T23:56:21.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.973+0000 7f3ec5600700 1 -- 172.21.15.12:0/2132954429 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3ec0024070 con 0x7f3ec81071c0 2024-03-17T23:56:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.974+0000 7f3ec5600700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3eb00a29a0 0x7f3eb00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.974+0000 7f3ec5600700 1 -- 172.21.15.12:0/2132954429 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f3ec000f020 con 0x7f3ec81071c0 2024-03-17T23:56:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.975+0000 7f3ecd000700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3eb00a29a0 0x7f3eb00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.975+0000 7f3ecd000700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3eb00a29a0 0x7f3eb00a4e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009680 tx=0x7f3ebc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:21.978+0000 7f3ec5600700 1 -- 172.21.15.12:0/2132954429 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3ec002b020 con 0x7f3ec81071c0 2024-03-17T23:56:22.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:21 smithi079 ceph-mon[36241]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.245+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2132954429 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f3ec81abd10 con 0x7f3ec81071c0 2024-03-17T23:56:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.245+0000 7f3ec5600700 1 -- 172.21.15.12:0/2132954429 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3ec0007d30 con 0x7f3ec81071c0 2024-03-17T23:56:22.246 INFO:teuthology.orchestra.run.smithi012.stdout:197568495675 2024-03-17T23:56:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.248+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2132954429 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3eb00a29a0 msgr2=0x7f3eb00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.248+0000 7f3ecfa00700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3eb00a29a0 0x7f3eb00a4e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009680 tx=0x7f3ebc009260 comp rx=0 tx=0).stop 2024-03-17T23:56:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.248+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2132954429 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ec81071c0 msgr2=0x7f3ec81a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.248+0000 7f3ecfa00700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ec81071c0 0x7f3ec81a3890 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000b070 tx=0x7f3ec0004c80 comp rx=0 tx=0).stop 2024-03-17T23:56:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.248+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2132954429 shutdown_connections 2024-03-17T23:56:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.248+0000 7f3ecfa00700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3eb00a29a0 0x7f3eb00a4e60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.248+0000 7f3ecfa00700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ec81071c0 0x7f3ec81a3890 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.248+0000 7f3ecfa00700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec806a4c0 0x7f3ec819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.248+0000 7f3ecfa00700 1 --2- 172.21.15.12:0/2132954429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ec8104cd0 0x7f3ec819f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.248+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2132954429 >> 172.21.15.12:0/2132954429 conn(0x7f3ec8078100 msgr2=0x7f3ec8103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.249+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2132954429 shutdown_connections 2024-03-17T23:56:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:22.249+0000 7f3ecfa00700 1 -- 172.21.15.12:0/2132954429 wait complete. 2024-03-17T23:56:22.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:22 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2132954429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-17T23:56:22.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:22 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2132954429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-17T23:56:23.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:22 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2132954429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-17T23:56:23.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:23 smithi012 ceph-mon[31541]: pgmap v366: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:56:23.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:23 smithi191 ceph-mon[36708]: pgmap v366: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:56:24.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:23 smithi079 ceph-mon[36241]: pgmap v366: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:56:25.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:25 smithi191 ceph-mon[36708]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-17T23:56:25.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:25 smithi012 ceph-mon[31541]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-17T23:56:26.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:25 smithi079 ceph-mon[36241]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-17T23:56:27.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.477+0000 7f75b5600700 1 -- 172.21.15.12:0/1326263402 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b0108300 msgr2=0x7f75b01086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:27.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.477+0000 7f75b5600700 1 --2- 172.21.15.12:0/1326263402 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b0108300 0x7f75b01086e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f75a00099c0 tx=0x7f75a0009cd0 comp rx=0 tx=0).stop 2024-03-17T23:56:27.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.478+0000 7f75b5600700 1 -- 172.21.15.12:0/1326263402 shutdown_connections 2024-03-17T23:56:27.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.478+0000 7f75b5600700 1 --2- 172.21.15.12:0/1326263402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75b0102ce0 0x7f75b010f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:27.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.478+0000 7f75b5600700 1 --2- 172.21.15.12:0/1326263402 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b0108300 0x7f75b01086e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:27.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.478+0000 7f75b5600700 1 --2- 172.21.15.12:0/1326263402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b0102320 0x7f75b01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:27.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.478+0000 7f75b5600700 1 -- 172.21.15.12:0/1326263402 >> 172.21.15.12:0/1326263402 conn(0x7f75b0079400 msgr2=0x7f75b0079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:27.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.478+0000 7f75b5600700 1 -- 172.21.15.12:0/1326263402 shutdown_connections 2024-03-17T23:56:27.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.478+0000 7f75b5600700 1 -- 172.21.15.12:0/1326263402 wait complete. 2024-03-17T23:56:27.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.479+0000 7f75b5600700 1 Processor -- start 2024-03-17T23:56:27.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.479+0000 7f75b5600700 1 -- start start 2024-03-17T23:56:27.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.480+0000 7f75b5600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b0102320 0x7f75b01a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:27.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.480+0000 7f75b5600700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b0102ce0 0x7f75b01a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:27.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.480+0000 7f75b5600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75b0108300 0x7f75b019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:27.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.480+0000 7f75b5600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f75b01125d0 con 0x7f75b0108300 2024-03-17T23:56:27.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.480+0000 7f75b5600700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f75b0112450 con 0x7f75b0102ce0 2024-03-17T23:56:27.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.480+0000 7f75b5600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f75b0112750 con 0x7f75b0102320 2024-03-17T23:56:27.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.480+0000 7f75aea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b0102320 0x7f75b01a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:27.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.480+0000 7f75aea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b0102320 0x7f75b01a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:36212/0 (socket says 172.21.15.12:36212) 2024-03-17T23:56:27.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.480+0000 7f75aea00700 1 -- 172.21.15.12:0/4116457523 learned_addr learned my addr 172.21.15.12:0/4116457523 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:27.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.480+0000 7f75ae000700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b0102ce0 0x7f75b01a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:27.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.480+0000 7f75af400700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75b0108300 0x7f75b019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:27.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.481+0000 7f75aea00700 1 -- 172.21.15.12:0/4116457523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b0102ce0 msgr2=0x7f75b01a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:27.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.481+0000 7f75aea00700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b0102ce0 0x7f75b01a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:27.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.481+0000 7f75aea00700 1 -- 172.21.15.12:0/4116457523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75b0108300 msgr2=0x7f75b019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:27.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.481+0000 7f75aea00700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75b0108300 0x7f75b019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:27.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.481+0000 7f75aea00700 1 -- 172.21.15.12:0/4116457523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b019b1c0 con 0x7f75b0102320 2024-03-17T23:56:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.481+0000 7f75af400700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75b0108300 0x7f75b019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:56:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.481+0000 7f75ae000700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b0102ce0 0x7f75b01a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.481+0000 7f75aea00700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b0102320 0x7f75b01a08a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f75a0005910 tx=0x7f75a000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.481+0000 7f75a7400700 1 -- 172.21.15.12:0/4116457523 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0027070 con 0x7f75b0102320 2024-03-17T23:56:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.482+0000 7f75a7400700 1 -- 172.21.15.12:0/4116457523 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a0003680 con 0x7f75b0102320 2024-03-17T23:56:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.482+0000 7f75a7400700 1 -- 172.21.15.12:0/4116457523 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a00173f0 con 0x7f75b0102320 2024-03-17T23:56:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.482+0000 7f75b5600700 1 -- 172.21.15.12:0/4116457523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b019b450 con 0x7f75b0102320 2024-03-17T23:56:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.482+0000 7f75b5600700 1 -- 172.21.15.12:0/4116457523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b01a7770 con 0x7f75b0102320 2024-03-17T23:56:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.483+0000 7f75b5600700 1 -- 172.21.15.12:0/4116457523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b01039b0 con 0x7f75b0102320 2024-03-17T23:56:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.485+0000 7f75a7400700 1 -- 172.21.15.12:0/4116457523 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f75a000fcd0 con 0x7f75b0102320 2024-03-17T23:56:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.487+0000 7f75a7400700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f75940a2920 0x7f75940a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.487+0000 7f75a7400700 1 -- 172.21.15.12:0/4116457523 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f75a00cf9a0 con 0x7f75b0102320 2024-03-17T23:56:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.487+0000 7f75ae000700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f75940a2920 0x7f75940a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:27.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.488+0000 7f75ae000700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f75940a2920 0x7f75940a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f75b0076640 tx=0x7f759c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:27.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.491+0000 7f75a7400700 1 -- 172.21.15.12:0/4116457523 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f75a0013070 con 0x7f75b0102320 2024-03-17T23:56:27.280 INFO:teuthology.orchestra.run.smithi012.stdout:219043332150 2024-03-17T23:56:27.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.739+0000 7f75b5600700 1 -- 172.21.15.12:0/4116457523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f75b01a7940 con 0x7f75b0102320 2024-03-17T23:56:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.740+0000 7f75a7400700 1 -- 172.21.15.12:0/4116457523 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f75a009c5b0 con 0x7f75b0102320 2024-03-17T23:56:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.743+0000 7f75b5600700 1 -- 172.21.15.12:0/4116457523 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f75940a2920 msgr2=0x7f75940a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.743+0000 7f75b5600700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f75940a2920 0x7f75940a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f75b0076640 tx=0x7f759c009260 comp rx=0 tx=0).stop 2024-03-17T23:56:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.743+0000 7f75b5600700 1 -- 172.21.15.12:0/4116457523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b0102320 msgr2=0x7f75b01a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.743+0000 7f75b5600700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b0102320 0x7f75b01a08a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f75a0005910 tx=0x7f75a000f690 comp rx=0 tx=0).stop 2024-03-17T23:56:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.743+0000 7f75b5600700 1 -- 172.21.15.12:0/4116457523 shutdown_connections 2024-03-17T23:56:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.743+0000 7f75b5600700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f75940a2920 0x7f75940a4de0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.743+0000 7f75b5600700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75b0108300 0x7f75b019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.743+0000 7f75b5600700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b0102320 0x7f75b01a08a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.743+0000 7f75b5600700 1 --2- 172.21.15.12:0/4116457523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75b0102ce0 0x7f75b01a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.743+0000 7f75b5600700 1 -- 172.21.15.12:0/4116457523 >> 172.21.15.12:0/4116457523 conn(0x7f75b0079400 msgr2=0x7f75b010d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.743+0000 7f75b5600700 1 -- 172.21.15.12:0/4116457523 shutdown_connections 2024-03-17T23:56:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:26.743+0000 7f75b5600700 1 -- 172.21.15.12:0/4116457523 wait complete. 2024-03-17T23:56:27.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:27 smithi191 ceph-mon[36708]: pgmap v368: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-17T23:56:27.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:27 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/4116457523' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-17T23:56:27.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:27 smithi012 ceph-mon[31541]: pgmap v368: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-17T23:56:27.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:27 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4116457523' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-17T23:56:28.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:27 smithi079 ceph-mon[36241]: pgmap v368: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-17T23:56:28.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:27 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/4116457523' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-17T23:56:28.660 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495672 got 197568495675 for osd.6 2024-03-17T23:56:28.660 DEBUG:teuthology.parallel:result is None 2024-03-17T23:56:29.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.401+0000 7efd58c00700 1 -- 172.21.15.12:0/483030157 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd54102510 msgr2=0x7efd541028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.401+0000 7efd58c00700 1 --2- 172.21.15.12:0/483030157 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd54102510 0x7efd541028f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efd44009990 tx=0x7efd44009ca0 comp rx=0 tx=0).stop 2024-03-17T23:56:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.401+0000 7efd58c00700 1 -- 172.21.15.12:0/483030157 shutdown_connections 2024-03-17T23:56:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.401+0000 7efd58c00700 1 --2- 172.21.15.12:0/483030157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd541075f0 0x7efd5410f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.401+0000 7efd58c00700 1 --2- 172.21.15.12:0/483030157 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd54102510 0x7efd541028f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.401+0000 7efd58c00700 1 --2- 172.21.15.12:0/483030157 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd54102ec0 0x7efd54106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.401+0000 7efd58c00700 1 -- 172.21.15.12:0/483030157 >> 172.21.15.12:0/483030157 conn(0x7efd540fdd80 msgr2=0x7efd541001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.401+0000 7efd58c00700 1 -- 172.21.15.12:0/483030157 shutdown_connections 2024-03-17T23:56:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.401+0000 7efd58c00700 1 -- 172.21.15.12:0/483030157 wait complete. 2024-03-17T23:56:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.403+0000 7efd58c00700 1 Processor -- start 2024-03-17T23:56:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.403+0000 7efd58c00700 1 -- start start 2024-03-17T23:56:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.404+0000 7efd58c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd54102510 0x7efd541a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.404+0000 7efd58c00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd54102ec0 0x7efd541a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.404+0000 7efd58c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd541075f0 0x7efd5419aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.404+0000 7efd58c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efd54112620 con 0x7efd54102510 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.404+0000 7efd58c00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efd541124a0 con 0x7efd54102ec0 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.404+0000 7efd58c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7efd541127a0 con 0x7efd541075f0 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.404+0000 7efd51400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd54102ec0 0x7efd541a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.404+0000 7efd51400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd54102ec0 0x7efd541a0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:41862/0 (socket says 172.21.15.12:41862) 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.404+0000 7efd51400700 1 -- 172.21.15.12:0/1785477980 learned_addr learned my addr 172.21.15.12:0/1785477980 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.404+0000 7efd52800700 1 --2- 172.21.15.12:0/1785477980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd541075f0 0x7efd5419aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.405+0000 7efd51400700 1 -- 172.21.15.12:0/1785477980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd541075f0 msgr2=0x7efd5419aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.405+0000 7efd51400700 1 --2- 172.21.15.12:0/1785477980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd541075f0 0x7efd5419aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.405+0000 7efd51400700 1 -- 172.21.15.12:0/1785477980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd54102510 msgr2=0x7efd541a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.405+0000 7efd51400700 1 --2- 172.21.15.12:0/1785477980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd54102510 0x7efd541a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.405+0000 7efd51400700 1 -- 172.21.15.12:0/1785477980 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5419b220 con 0x7efd54102ec0 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.405+0000 7efd51400700 1 --2- 172.21.15.12:0/1785477980 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd54102ec0 0x7efd541a0e40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efd4000e970 tx=0x7efd4000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.405+0000 7efd4aa00700 1 -- 172.21.15.12:0/1785477980 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4000cbd0 con 0x7efd54102ec0 2024-03-17T23:56:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.405+0000 7efd4aa00700 1 -- 172.21.15.12:0/1785477980 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd4000cd70 con 0x7efd54102ec0 2024-03-17T23:56:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.405+0000 7efd58c00700 1 -- 172.21.15.12:0/1785477980 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd5419b510 con 0x7efd54102ec0 2024-03-17T23:56:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.406+0000 7efd4aa00700 1 -- 172.21.15.12:0/1785477980 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd40005660 con 0x7efd54102ec0 2024-03-17T23:56:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.406+0000 7efd58c00700 1 -- 172.21.15.12:0/1785477980 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd541a77d0 con 0x7efd54102ec0 2024-03-17T23:56:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.407+0000 7efd58c00700 1 -- 172.21.15.12:0/1785477980 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd54068f90 con 0x7efd54102ec0 2024-03-17T23:56:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.409+0000 7efd4aa00700 1 -- 172.21.15.12:0/1785477980 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7efd40004750 con 0x7efd54102ec0 2024-03-17T23:56:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.410+0000 7efd4aa00700 1 --2- 172.21.15.12:0/1785477980 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7efd380a29a0 0x7efd380a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.410+0000 7efd4aa00700 1 -- 172.21.15.12:0/1785477980 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7efd400cf1c0 con 0x7efd54102ec0 2024-03-17T23:56:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.410+0000 7efd51e00700 1 --2- 172.21.15.12:0/1785477980 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7efd380a29a0 0x7efd380a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.411+0000 7efd51e00700 1 --2- 172.21.15.12:0/1785477980 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7efd380a29a0 0x7efd380a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7efd440098e0 tx=0x7efd44005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.419+0000 7efd4aa00700 1 -- 172.21.15.12:0/1785477980 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd4009bd40 con 0x7efd54102ec0 2024-03-17T23:56:29.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.656+0000 7efd58c00700 1 -- 172.21.15.12:0/1785477980 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7efd5419bff0 con 0x7efd54102ec0 2024-03-17T23:56:29.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.656+0000 7efd4aa00700 1 -- 172.21.15.12:0/1785477980 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7efd40097020 con 0x7efd54102ec0 2024-03-17T23:56:29.657 INFO:teuthology.orchestra.run.smithi012.stdout:141733920841 2024-03-17T23:56:29.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.659+0000 7efd58c00700 1 -- 172.21.15.12:0/1785477980 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7efd380a29a0 msgr2=0x7efd380a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:29.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.659+0000 7efd58c00700 1 --2- 172.21.15.12:0/1785477980 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7efd380a29a0 0x7efd380a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7efd440098e0 tx=0x7efd44005fb0 comp rx=0 tx=0).stop 2024-03-17T23:56:29.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.659+0000 7efd58c00700 1 -- 172.21.15.12:0/1785477980 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd54102ec0 msgr2=0x7efd541a0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:29.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.659+0000 7efd58c00700 1 --2- 172.21.15.12:0/1785477980 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd54102ec0 0x7efd541a0e40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efd4000e970 tx=0x7efd4000ec80 comp rx=0 tx=0).stop 2024-03-17T23:56:29.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.659+0000 7efd58c00700 1 -- 172.21.15.12:0/1785477980 shutdown_connections 2024-03-17T23:56:29.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.659+0000 7efd58c00700 1 --2- 172.21.15.12:0/1785477980 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7efd380a29a0 0x7efd380a4e60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:29.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.659+0000 7efd58c00700 1 --2- 172.21.15.12:0/1785477980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd541075f0 0x7efd5419aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:29.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.659+0000 7efd58c00700 1 --2- 172.21.15.12:0/1785477980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd54102510 0x7efd541a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:29.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.659+0000 7efd58c00700 1 --2- 172.21.15.12:0/1785477980 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd54102ec0 0x7efd541a0e40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:29.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.659+0000 7efd58c00700 1 -- 172.21.15.12:0/1785477980 >> 172.21.15.12:0/1785477980 conn(0x7efd540fdd80 msgr2=0x7efd5410e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:29.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.660+0000 7efd58c00700 1 -- 172.21.15.12:0/1785477980 shutdown_connections 2024-03-17T23:56:29.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.660+0000 7efd58c00700 1 -- 172.21.15.12:0/1785477980 wait complete. 2024-03-17T23:56:29.793 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:29 smithi012 ceph-mon[31541]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-17T23:56:29.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:29 smithi191 ceph-mon[36708]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-17T23:56:30.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:29 smithi079 ceph-mon[36241]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-17T23:56:30.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:30 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1785477980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-17T23:56:30.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:30 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4118859916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-17T23:56:30.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.512+0000 7fadb6c00700 1 -- 172.21.15.12:0/3114502405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb0102410 msgr2=0x7fadb01027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:30.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.512+0000 7fadb6c00700 1 --2- 172.21.15.12:0/3114502405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb0102410 0x7fadb01027f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fada4009990 tx=0x7fada4009ca0 comp rx=0 tx=0).stop 2024-03-17T23:56:30.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.513+0000 7fadb6c00700 1 -- 172.21.15.12:0/3114502405 shutdown_connections 2024-03-17T23:56:30.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.513+0000 7fadb6c00700 1 --2- 172.21.15.12:0/3114502405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb01074f0 0x7fadb010f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:30.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.513+0000 7fadb6c00700 1 --2- 172.21.15.12:0/3114502405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb0102410 0x7fadb01027f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:30.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.513+0000 7fadb6c00700 1 --2- 172.21.15.12:0/3114502405 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fadb0102dc0 0x7fadb0106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:30.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.513+0000 7fadb6c00700 1 -- 172.21.15.12:0/3114502405 >> 172.21.15.12:0/3114502405 conn(0x7fadb00fdca0 msgr2=0x7fadb01000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:30.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.513+0000 7fadb6c00700 1 -- 172.21.15.12:0/3114502405 shutdown_connections 2024-03-17T23:56:30.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.513+0000 7fadb6c00700 1 -- 172.21.15.12:0/3114502405 wait complete. 2024-03-17T23:56:30.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.514+0000 7fadb6c00700 1 Processor -- start 2024-03-17T23:56:30.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.514+0000 7fadb6c00700 1 -- start start 2024-03-17T23:56:30.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.515+0000 7fadb6c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fadb0102410 0x7fadb019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:30.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.515+0000 7fadb6c00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb0102dc0 0x7fadb019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:30.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.515+0000 7fadb6c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb01074f0 0x7fadb0198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.515+0000 7fadb6c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fadb0112540 con 0x7fadb01074f0 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.515+0000 7fadb6c00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fadb01123c0 con 0x7fadb0102dc0 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.515+0000 7fadb6c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fadb01126c0 con 0x7fadb0102410 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.515+0000 7fadb5800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fadb0102410 0x7fadb019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.515+0000 7fadb5800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fadb0102410 0x7fadb019e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:36256/0 (socket says 172.21.15.12:36256) 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.515+0000 7fadb5800700 1 -- 172.21.15.12:0/4118859916 learned_addr learned my addr 172.21.15.12:0/4118859916 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.515+0000 7fadb4e00700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb0102dc0 0x7fadb019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.515+0000 7fadb6200700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb01074f0 0x7fadb0198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.516+0000 7fadb5800700 1 -- 172.21.15.12:0/4118859916 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb0102dc0 msgr2=0x7fadb019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.516+0000 7fadb5800700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb0102dc0 0x7fadb019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.516+0000 7fadb5800700 1 -- 172.21.15.12:0/4118859916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb01074f0 msgr2=0x7fadb0198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.516+0000 7fadb5800700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb01074f0 0x7fadb0198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.516+0000 7fadb5800700 1 -- 172.21.15.12:0/4118859916 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb0199000 con 0x7fadb0102410 2024-03-17T23:56:30.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.516+0000 7fadb5800700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fadb0102410 0x7fadb019e6a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fada4000c00 tx=0x7fada40057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:30.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.516+0000 7fadae000700 1 -- 172.21.15.12:0/4118859916 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4027070 con 0x7fadb0102410 2024-03-17T23:56:30.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.516+0000 7fadae000700 1 -- 172.21.15.12:0/4118859916 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fada4022470 con 0x7fadb0102410 2024-03-17T23:56:30.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.516+0000 7fadae000700 1 -- 172.21.15.12:0/4118859916 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4017650 con 0x7fadb0102410 2024-03-17T23:56:30.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.516+0000 7fadb6c00700 1 -- 172.21.15.12:0/4118859916 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadb0199290 con 0x7fadb0102410 2024-03-17T23:56:30.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.517+0000 7fadb6c00700 1 -- 172.21.15.12:0/4118859916 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadb01a5500 con 0x7fadb0102410 2024-03-17T23:56:30.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.518+0000 7fadb6c00700 1 -- 172.21.15.12:0/4118859916 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadb01038f0 con 0x7fadb0102410 2024-03-17T23:56:30.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.520+0000 7fadae000700 1 -- 172.21.15.12:0/4118859916 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fada402a070 con 0x7fadb0102410 2024-03-17T23:56:30.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.522+0000 7fadae000700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fad980a2920 0x7fad980a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:30.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.522+0000 7fadae000700 1 -- 172.21.15.12:0/4118859916 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fada40cfd50 con 0x7fadb0102410 2024-03-17T23:56:30.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.522+0000 7fadb4e00700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fad980a2920 0x7fad980a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:30.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.523+0000 7fadb4e00700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fad980a2920 0x7fad980a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fadb0108aa0 tx=0x7fad9c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:30.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.525+0000 7fadae000700 1 -- 172.21.15.12:0/4118859916 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fada4013070 con 0x7fadb0102410 2024-03-17T23:56:30.925 INFO:teuthology.orchestra.run.smithi012.stdout:287762808875 2024-03-17T23:56:30.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.791+0000 7fadb6c00700 1 -- 172.21.15.12:0/4118859916 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fadb0199d50 con 0x7fadb0102410 2024-03-17T23:56:30.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.792+0000 7fadae000700 1 -- 172.21.15.12:0/4118859916 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fada409c960 con 0x7fadb0102410 2024-03-17T23:56:30.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.795+0000 7fadb6c00700 1 -- 172.21.15.12:0/4118859916 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fad980a2920 msgr2=0x7fad980a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:30.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.795+0000 7fadb6c00700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fad980a2920 0x7fad980a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fadb0108aa0 tx=0x7fad9c009260 comp rx=0 tx=0).stop 2024-03-17T23:56:30.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.795+0000 7fadb6c00700 1 -- 172.21.15.12:0/4118859916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fadb0102410 msgr2=0x7fadb019e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:30.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.795+0000 7fadb6c00700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fadb0102410 0x7fadb019e6a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fada4000c00 tx=0x7fada40057c0 comp rx=0 tx=0).stop 2024-03-17T23:56:30.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.795+0000 7fadb6c00700 1 -- 172.21.15.12:0/4118859916 shutdown_connections 2024-03-17T23:56:30.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.795+0000 7fadb6c00700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fad980a2920 0x7fad980a4de0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:30.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.795+0000 7fadb6c00700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadb01074f0 0x7fadb0198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:30.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.795+0000 7fadb6c00700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fadb0102410 0x7fadb019e6a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:30.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.795+0000 7fadb6c00700 1 --2- 172.21.15.12:0/4118859916 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb0102dc0 0x7fadb019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:30.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.795+0000 7fadb6c00700 1 -- 172.21.15.12:0/4118859916 >> 172.21.15.12:0/4118859916 conn(0x7fadb00fdca0 msgr2=0x7fadb00fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:30.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.795+0000 7fadb6c00700 1 -- 172.21.15.12:0/4118859916 shutdown_connections 2024-03-17T23:56:30.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:29.796+0000 7fadb6c00700 1 -- 172.21.15.12:0/4118859916 wait complete. 2024-03-17T23:56:31.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:30 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1785477980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-17T23:56:31.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:30 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/4118859916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-17T23:56:31.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:30 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1785477980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-17T23:56:31.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:30 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/4118859916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-17T23:56:31.911 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:31 smithi012 ceph-mon[31541]: pgmap v370: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:32.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:31 smithi079 ceph-mon[36241]: pgmap v370: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:32.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:31 smithi191 ceph-mon[36708]: pgmap v370: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:32.821 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:33.728 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:33 smithi012 ceph-mon[31541]: pgmap v371: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:56:33.763 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332146 got 219043332150 for osd.7 2024-03-17T23:56:33.763 DEBUG:teuthology.parallel:result is None 2024-03-17T23:56:34.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:33 smithi079 ceph-mon[36241]: pgmap v371: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:56:34.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:33 smithi191 ceph-mon[36708]: pgmap v371: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:56:34.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.282+0000 7f5813a00700 1 -- 172.21.15.12:0/1243160866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c074500 msgr2=0x7f580c0748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:34.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.282+0000 7f5813a00700 1 --2- 172.21.15.12:0/1243160866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c074500 0x7f580c0748e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f58000099c0 tx=0x7f5800009cd0 comp rx=0 tx=0).stop 2024-03-17T23:56:34.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.282+0000 7f5813a00700 1 -- 172.21.15.12:0/1243160866 shutdown_connections 2024-03-17T23:56:34.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.282+0000 7f5813a00700 1 --2- 172.21.15.12:0/1243160866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f580c1134a0 0x7f580c115890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:34.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.282+0000 7f5813a00700 1 --2- 172.21.15.12:0/1243160866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c074500 0x7f580c0748e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:34.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.282+0000 7f5813a00700 1 --2- 172.21.15.12:0/1243160866 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f580c074e20 0x7f580c112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:34.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.282+0000 7f5813a00700 1 -- 172.21.15.12:0/1243160866 >> 172.21.15.12:0/1243160866 conn(0x7f580c079430 msgr2=0x7f580c079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:34.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.283+0000 7f5813a00700 1 -- 172.21.15.12:0/1243160866 shutdown_connections 2024-03-17T23:56:34.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.283+0000 7f5813a00700 1 -- 172.21.15.12:0/1243160866 wait complete. 2024-03-17T23:56:34.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.284+0000 7f5813a00700 1 Processor -- start 2024-03-17T23:56:34.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.285+0000 7f5813a00700 1 -- start start 2024-03-17T23:56:34.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.285+0000 7f5813a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f580c074500 0x7f580c19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:34.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.285+0000 7f5813a00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f580c074e20 0x7f580c19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:34.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.285+0000 7f5813a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c1134a0 0x7f580c1a3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:34.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.285+0000 7f5813a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f580c118ce0 con 0x7f580c074500 2024-03-17T23:56:34.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.285+0000 7f5813a00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f580c118b60 con 0x7f580c074e20 2024-03-17T23:56:34.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.285+0000 7f5813a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f580c118e60 con 0x7f580c1134a0 2024-03-17T23:56:34.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.286+0000 7f5811000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f580c074500 0x7f580c19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:34.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.286+0000 7f580be00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f580c074e20 0x7f580c19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:34.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.286+0000 7f580be00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f580c074e20 0x7f580c19f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:41122/0 (socket says 172.21.15.12:41122) 2024-03-17T23:56:34.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.286+0000 7f580be00700 1 -- 172.21.15.12:0/2395973322 learned_addr learned my addr 172.21.15.12:0/2395973322 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:34.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.286+0000 7f5811a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c1134a0 0x7f580c1a3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:34.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.286+0000 7f5811000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f580c074500 0x7f580c19f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52308/0 (socket says 172.21.15.12:52308) 2024-03-17T23:56:34.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.286+0000 7f5811a00700 1 -- 172.21.15.12:0/2395973322 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f580c074e20 msgr2=0x7f580c19f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:34.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.286+0000 7f5811a00700 1 --2- 172.21.15.12:0/2395973322 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f580c074e20 0x7f580c19f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.286+0000 7f5811a00700 1 -- 172.21.15.12:0/2395973322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f580c074500 msgr2=0x7f580c19f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.286+0000 7f5811a00700 1 --2- 172.21.15.12:0/2395973322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f580c074500 0x7f580c19f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.286+0000 7f5811a00700 1 -- 172.21.15.12:0/2395973322 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f580c1a4140 con 0x7f580c1134a0 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.286+0000 7f580be00700 1 --2- 172.21.15.12:0/2395973322 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f580c074e20 0x7f580c19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.286+0000 7f5811a00700 1 --2- 172.21.15.12:0/2395973322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c1134a0 0x7f580c1a3a00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f580400efc0 tx=0x7f580400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.287+0000 7f5809600700 1 -- 172.21.15.12:0/2395973322 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58040092e0 con 0x7f580c1134a0 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.287+0000 7f5809600700 1 -- 172.21.15.12:0/2395973322 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5804004bb0 con 0x7f580c1134a0 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.287+0000 7f5813a00700 1 -- 172.21.15.12:0/2395973322 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f580c1a43d0 con 0x7f580c1134a0 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.287+0000 7f5809600700 1 -- 172.21.15.12:0/2395973322 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5804004280 con 0x7f580c1134a0 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.287+0000 7f5813a00700 1 -- 172.21.15.12:0/2395973322 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f580c1abcb0 con 0x7f580c1134a0 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.288+0000 7f5813a00700 1 -- 172.21.15.12:0/2395973322 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f580c068f90 con 0x7f580c1134a0 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.292+0000 7f5809600700 1 -- 172.21.15.12:0/2395973322 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f580401b030 con 0x7f580c1134a0 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.294+0000 7f5809600700 1 --2- 172.21.15.12:0/2395973322 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f57f40a29a0 0x7f57f40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.294+0000 7f5809600700 1 -- 172.21.15.12:0/2395973322 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f58040ce9a0 con 0x7f580c1134a0 2024-03-17T23:56:34.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.294+0000 7f5811000700 1 --2- 172.21.15.12:0/2395973322 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f57f40a29a0 0x7f57f40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:34.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.295+0000 7f5811000700 1 --2- 172.21.15.12:0/2395973322 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f57f40a29a0 0x7f57f40a4e60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5800000c00 tx=0x7f5800005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:34.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.298+0000 7f5809600700 1 -- 172.21.15.12:0/2395973322 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5804022020 con 0x7f580c1134a0 2024-03-17T23:56:34.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.566+0000 7f5813a00700 1 -- 172.21.15.12:0/2395973322 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f580c1abf40 con 0x7f580c1134a0 2024-03-17T23:56:34.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.567+0000 7f5809600700 1 -- 172.21.15.12:0/2395973322 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5804010040 con 0x7f580c1134a0 2024-03-17T23:56:34.568 INFO:teuthology.orchestra.run.smithi012.stdout:253403070514 2024-03-17T23:56:34.570 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920837 got 141733920841 for osd.4 2024-03-17T23:56:34.570 DEBUG:teuthology.parallel:result is None 2024-03-17T23:56:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.570+0000 7f5813a00700 1 -- 172.21.15.12:0/2395973322 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f57f40a29a0 msgr2=0x7f57f40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.570+0000 7f5813a00700 1 --2- 172.21.15.12:0/2395973322 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f57f40a29a0 0x7f57f40a4e60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5800000c00 tx=0x7f5800005fb0 comp rx=0 tx=0).stop 2024-03-17T23:56:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.570+0000 7f5813a00700 1 -- 172.21.15.12:0/2395973322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c1134a0 msgr2=0x7f580c1a3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.570+0000 7f5813a00700 1 --2- 172.21.15.12:0/2395973322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c1134a0 0x7f580c1a3a00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f580400efc0 tx=0x7f580400c3a0 comp rx=0 tx=0).stop 2024-03-17T23:56:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.570+0000 7f5813a00700 1 -- 172.21.15.12:0/2395973322 shutdown_connections 2024-03-17T23:56:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.570+0000 7f5813a00700 1 --2- 172.21.15.12:0/2395973322 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f57f40a29a0 0x7f57f40a4e60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.570+0000 7f5813a00700 1 --2- 172.21.15.12:0/2395973322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c1134a0 0x7f580c1a3a00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.570+0000 7f5813a00700 1 --2- 172.21.15.12:0/2395973322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f580c074500 0x7f580c19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.570+0000 7f5813a00700 1 --2- 172.21.15.12:0/2395973322 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f580c074e20 0x7f580c19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.570+0000 7f5813a00700 1 -- 172.21.15.12:0/2395973322 >> 172.21.15.12:0/2395973322 conn(0x7f580c079430 msgr2=0x7f580c112260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.570+0000 7f5813a00700 1 -- 172.21.15.12:0/2395973322 shutdown_connections 2024-03-17T23:56:34.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.570+0000 7f5813a00700 1 -- 172.21.15.12:0/2395973322 wait complete. 2024-03-17T23:56:34.790 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:34 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3467255516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-17T23:56:34.790 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:34 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2395973322' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-17T23:56:35.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:34 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3467255516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-17T23:56:35.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:34 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2395973322' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-17T23:56:35.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:34 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3467255516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-17T23:56:35.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:34 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2395973322' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-17T23:56:35.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:35 smithi012 ceph-mon[31541]: pgmap v372: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:36.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:35 smithi079 ceph-mon[36241]: pgmap v372: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:36.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:35 smithi191 ceph-mon[36708]: pgmap v372: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:36.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.898+0000 7f437d800700 1 -- 172.21.15.12:0/4287600400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437810eb90 msgr2=0x7f4378110fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:36.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.898+0000 7f437d800700 1 --2- 172.21.15.12:0/4287600400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437810eb90 0x7f4378110fa0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4370009f90 tx=0x7f437000b330 comp rx=0 tx=0).stop 2024-03-17T23:56:36.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.899+0000 7f437d800700 1 -- 172.21.15.12:0/4287600400 shutdown_connections 2024-03-17T23:56:36.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.899+0000 7f437d800700 1 --2- 172.21.15.12:0/4287600400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437810eb90 0x7f4378110fa0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:36.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.899+0000 7f437d800700 1 --2- 172.21.15.12:0/4287600400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43780746a0 0x7f4378074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:36.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.899+0000 7f437d800700 1 --2- 172.21.15.12:0/4287600400 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4378074fc0 0x7f437810e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:36.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.899+0000 7f437d800700 1 -- 172.21.15.12:0/4287600400 >> 172.21.15.12:0/4287600400 conn(0x7f43780fdda0 msgr2=0x7f43781001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:36.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.899+0000 7f437d800700 1 -- 172.21.15.12:0/4287600400 shutdown_connections 2024-03-17T23:56:36.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.899+0000 7f437d800700 1 -- 172.21.15.12:0/4287600400 wait complete. 2024-03-17T23:56:36.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.900+0000 7f437d800700 1 Processor -- start 2024-03-17T23:56:36.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.900+0000 7f437d800700 1 -- start start 2024-03-17T23:56:36.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f437d800700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43780746a0 0x7f43781a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:36.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f437d800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4378074fc0 0x7f43781a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:36.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f437d800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437810eb90 0x7f437819aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:36.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f437d800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4378114860 con 0x7f4378074fc0 2024-03-17T23:56:36.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f437d800700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f43781146e0 con 0x7f43780746a0 2024-03-17T23:56:36.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f437d800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f43781149e0 con 0x7f437810eb90 2024-03-17T23:56:36.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f4376a00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43780746a0 0x7f43781a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:36.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f4376a00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43780746a0 0x7f43781a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:41134/0 (socket says 172.21.15.12:41134) 2024-03-17T23:56:36.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f4376a00700 1 -- 172.21.15.12:0/1327766543 learned_addr learned my addr 172.21.15.12:0/1327766543 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:36.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f4376000700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4378074fc0 0x7f43781a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:36.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f4377400700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437810eb90 0x7f437819aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:36.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f4376a00700 1 -- 172.21.15.12:0/1327766543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437810eb90 msgr2=0x7f437819aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:36.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f4376a00700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437810eb90 0x7f437819aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:36.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f4376a00700 1 -- 172.21.15.12:0/1327766543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4378074fc0 msgr2=0x7f43781a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:36.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f4376a00700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4378074fc0 0x7f43781a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:36.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.901+0000 7f4376a00700 1 -- 172.21.15.12:0/1327766543 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437819b200 con 0x7f43780746a0 2024-03-17T23:56:36.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.902+0000 7f4377400700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437810eb90 0x7f437819aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:36.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.902+0000 7f4376a00700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43780746a0 0x7f43781a08e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f436800e9a0 tx=0x7f436800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:36.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.902+0000 7f436f400700 1 -- 172.21.15.12:0/1327766543 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436800c9c0 con 0x7f43780746a0 2024-03-17T23:56:36.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.902+0000 7f436f400700 1 -- 172.21.15.12:0/1327766543 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f436800cb60 con 0x7f43780746a0 2024-03-17T23:56:36.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.902+0000 7f437d800700 1 -- 172.21.15.12:0/1327766543 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f437819b4f0 con 0x7f43780746a0 2024-03-17T23:56:36.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.902+0000 7f436f400700 1 -- 172.21.15.12:0/1327766543 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4368018660 con 0x7f43780746a0 2024-03-17T23:56:36.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.902+0000 7f437d800700 1 -- 172.21.15.12:0/1327766543 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43781a77b0 con 0x7f43780746a0 2024-03-17T23:56:36.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.904+0000 7f437d800700 1 -- 172.21.15.12:0/1327766543 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4378068f90 con 0x7f43780746a0 2024-03-17T23:56:36.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.905+0000 7f436f400700 1 -- 172.21.15.12:0/1327766543 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f436800cd10 con 0x7f43780746a0 2024-03-17T23:56:36.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.906+0000 7f436f400700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f435c0a29a0 0x7f435c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:36.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.907+0000 7f436f400700 1 -- 172.21.15.12:0/1327766543 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f43680cf0d0 con 0x7f43780746a0 2024-03-17T23:56:36.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.907+0000 7f4376000700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f435c0a29a0 0x7f435c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:36.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.908+0000 7f4376000700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f435c0a29a0 0x7f435c0a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4378101940 tx=0x7f4364009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:36.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:36.915+0000 7f436f400700 1 -- 172.21.15.12:0/1327766543 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f436809bc50 con 0x7f43780746a0 2024-03-17T23:56:37.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.192+0000 7f437d800700 1 -- 172.21.15.12:0/1327766543 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f437802db70 con 0x7f43780746a0 2024-03-17T23:56:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.193+0000 7f436f400700 1 -- 172.21.15.12:0/1327766543 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f436809b550 con 0x7f43780746a0 2024-03-17T23:56:37.194 INFO:teuthology.orchestra.run.smithi012.stdout:107374182479 2024-03-17T23:56:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.182+0000 7f404ae00700 1 -- 172.21.15.12:0/2288539173 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4044101280 msgr2=0x7f404410f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.182+0000 7f404ae00700 1 --2- 172.21.15.12:0/2288539173 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4044101280 0x7f404410f1e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f403c00b0a0 tx=0x7f403c00b3b0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.182+0000 7f404ae00700 1 -- 172.21.15.12:0/2288539173 shutdown_connections 2024-03-17T23:56:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.182+0000 7f404ae00700 1 --2- 172.21.15.12:0/2288539173 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4044101280 0x7f404410f1e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.182+0000 7f404ae00700 1 --2- 172.21.15.12:0/2288539173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4044107340 0x7f4044107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.182+0000 7f404ae00700 1 --2- 172.21.15.12:0/2288539173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40441008c0 0x7f4044100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.182+0000 7f404ae00700 1 -- 172.21.15.12:0/2288539173 >> 172.21.15.12:0/2288539173 conn(0x7f40440777d0 msgr2=0x7f40440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.182+0000 7f404ae00700 1 -- 172.21.15.12:0/2288539173 shutdown_connections 2024-03-17T23:56:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.182+0000 7f404ae00700 1 -- 172.21.15.12:0/2288539173 wait complete. 2024-03-17T23:56:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.183+0000 7f404ae00700 1 Processor -- start 2024-03-17T23:56:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.184+0000 7f404ae00700 1 -- start start 2024-03-17T23:56:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.184+0000 7f404ae00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40441008c0 0x7f4044101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.184+0000 7f404ae00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4044101280 0x7f4044101fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.184+0000 7f404ae00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4044107340 0x7f4044102520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:37.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.184+0000 7f404ae00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4044112770 con 0x7f40441008c0 2024-03-17T23:56:37.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.184+0000 7f404ae00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f40441125f0 con 0x7f4044101280 2024-03-17T23:56:37.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.184+0000 7f404ae00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f40441128f0 con 0x7f4044107340 2024-03-17T23:56:37.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.185+0000 7f4043400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4044101280 0x7f4044101fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:37.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.185+0000 7f4043e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40441008c0 0x7f4044101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:37.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.185+0000 7f4043400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4044101280 0x7f4044101fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:41110/0 (socket says 172.21.15.12:41110) 2024-03-17T23:56:37.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.185+0000 7f4043400700 1 -- 172.21.15.12:0/3467255516 learned_addr learned my addr 172.21.15.12:0/3467255516 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:37.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.185+0000 7f4048e00700 1 --2- 172.21.15.12:0/3467255516 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4044107340 0x7f4044102520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:37.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.185+0000 7f4043400700 1 -- 172.21.15.12:0/3467255516 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4044107340 msgr2=0x7f4044102520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:37.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.185+0000 7f4043400700 1 --2- 172.21.15.12:0/3467255516 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4044107340 0x7f4044102520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.185+0000 7f4043400700 1 -- 172.21.15.12:0/3467255516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40441008c0 msgr2=0x7f4044101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:37.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.185+0000 7f4043400700 1 --2- 172.21.15.12:0/3467255516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40441008c0 0x7f4044101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.185+0000 7f4043400700 1 -- 172.21.15.12:0/3467255516 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40441a70b0 con 0x7f4044101280 2024-03-17T23:56:37.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.185+0000 7f4043400700 1 --2- 172.21.15.12:0/3467255516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4044101280 0x7f4044101fe0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f403000e9c0 tx=0x7f403000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:37.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.186+0000 7f4040c00700 1 -- 172.21.15.12:0/3467255516 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403000cbd0 con 0x7f4044101280 2024-03-17T23:56:37.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.186+0000 7f4040c00700 1 -- 172.21.15.12:0/3467255516 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f403000ee60 con 0x7f4044101280 2024-03-17T23:56:37.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.186+0000 7f4040c00700 1 -- 172.21.15.12:0/3467255516 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030005630 con 0x7f4044101280 2024-03-17T23:56:37.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.186+0000 7f404ae00700 1 -- 172.21.15.12:0/3467255516 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40441a73a0 con 0x7f4044101280 2024-03-17T23:56:37.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.186+0000 7f404ae00700 1 -- 172.21.15.12:0/3467255516 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40441a7890 con 0x7f4044101280 2024-03-17T23:56:37.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.187+0000 7f404ae00700 1 -- 172.21.15.12:0/3467255516 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4044103be0 con 0x7f4044101280 2024-03-17T23:56:37.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.191+0000 7f4040c00700 1 -- 172.21.15.12:0/3467255516 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4030004750 con 0x7f4044101280 2024-03-17T23:56:37.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.194+0000 7f4040c00700 1 --2- 172.21.15.12:0/3467255516 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f40280a29a0 0x7f40280a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:37.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.194+0000 7f4040c00700 1 -- 172.21.15.12:0/3467255516 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f40300cf3c0 con 0x7f4044101280 2024-03-17T23:56:37.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.195+0000 7f4043e00700 1 --2- 172.21.15.12:0/3467255516 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f40280a29a0 0x7f40280a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:37.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.195+0000 7f4040c00700 1 -- 172.21.15.12:0/3467255516 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f40300cf840 con 0x7f4044101280 2024-03-17T23:56:37.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.195+0000 7f4043e00700 1 --2- 172.21.15.12:0/3467255516 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f40280a29a0 0x7f40280a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4034009be0 tx=0x7f4034009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:37.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.463+0000 7f404ae00700 1 -- 172.21.15.12:0/3467255516 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f4044068f90 con 0x7f4044101280 2024-03-17T23:56:37.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.463+0000 7f4040c00700 1 -- 172.21.15.12:0/3467255516 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f403009bf40 con 0x7f4044101280 2024-03-17T23:56:37.199 INFO:teuthology.orchestra.run.smithi012.stdout:77309411413 2024-03-17T23:56:37.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.466+0000 7f404ae00700 1 -- 172.21.15.12:0/3467255516 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f40280a29a0 msgr2=0x7f40280a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:37.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.466+0000 7f404ae00700 1 --2- 172.21.15.12:0/3467255516 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f40280a29a0 0x7f40280a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4034009be0 tx=0x7f4034009260 comp rx=0 tx=0).stop 2024-03-17T23:56:37.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.466+0000 7f404ae00700 1 -- 172.21.15.12:0/3467255516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4044101280 msgr2=0x7f4044101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.466+0000 7f404ae00700 1 --2- 172.21.15.12:0/3467255516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4044101280 0x7f4044101fe0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f403000e9c0 tx=0x7f403000ecd0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.466+0000 7f404ae00700 1 -- 172.21.15.12:0/3467255516 shutdown_connections 2024-03-17T23:56:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.466+0000 7f404ae00700 1 --2- 172.21.15.12:0/3467255516 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f40280a29a0 0x7f40280a4e60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.466+0000 7f404ae00700 1 --2- 172.21.15.12:0/3467255516 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4044107340 0x7f4044102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.466+0000 7f404ae00700 1 --2- 172.21.15.12:0/3467255516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40441008c0 0x7f4044101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.466+0000 7f404ae00700 1 --2- 172.21.15.12:0/3467255516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4044101280 0x7f4044101fe0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.466+0000 7f404ae00700 1 -- 172.21.15.12:0/3467255516 >> 172.21.15.12:0/3467255516 conn(0x7f40440777d0 msgr2=0x7f4044105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.466+0000 7f404ae00700 1 -- 172.21.15.12:0/3467255516 shutdown_connections 2024-03-17T23:56:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:34.467+0000 7f404ae00700 1 -- 172.21.15.12:0/3467255516 wait complete. 2024-03-17T23:56:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.196+0000 7f437d800700 1 -- 172.21.15.12:0/1327766543 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f435c0a29a0 msgr2=0x7f435c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.196+0000 7f437d800700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f435c0a29a0 0x7f435c0a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4378101940 tx=0x7f4364009310 comp rx=0 tx=0).stop 2024-03-17T23:56:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.196+0000 7f437d800700 1 -- 172.21.15.12:0/1327766543 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43780746a0 msgr2=0x7f43781a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.196+0000 7f437d800700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43780746a0 0x7f43781a08e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f436800e9a0 tx=0x7f436800ecb0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.197+0000 7f437d800700 1 -- 172.21.15.12:0/1327766543 shutdown_connections 2024-03-17T23:56:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.197+0000 7f437d800700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f435c0a29a0 0x7f435c0a4e60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.197+0000 7f437d800700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437810eb90 0x7f437819aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.197+0000 7f437d800700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43780746a0 0x7f43781a08e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.197+0000 7f437d800700 1 --2- 172.21.15.12:0/1327766543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4378074fc0 0x7f43781a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.197+0000 7f437d800700 1 -- 172.21.15.12:0/1327766543 >> 172.21.15.12:0/1327766543 conn(0x7f43780fdda0 msgr2=0x7f43780fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.197+0000 7f437d800700 1 -- 172.21.15.12:0/1327766543 shutdown_connections 2024-03-17T23:56:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:37.197+0000 7f437d800700 1 -- 172.21.15.12:0/1327766543 wait complete. 2024-03-17T23:56:37.782 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:37 smithi012 ceph-mon[31541]: pgmap v373: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:56:37.782 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:37 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1327766543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-17T23:56:38.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:37 smithi079 ceph-mon[36241]: pgmap v373: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:56:38.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:37 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1327766543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-17T23:56:38.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:37 smithi191 ceph-mon[36708]: pgmap v373: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:56:38.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:37 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1327766543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-17T23:56:38.638 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808872 got 287762808875 for osd.9 2024-03-17T23:56:38.639 DEBUG:teuthology.parallel:result is None 2024-03-17T23:56:38.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.881+0000 7fd4dd600700 1 -- 172.21.15.12:0/756151007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4d8074e20 msgr2=0x7fd4d8112f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:38.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.881+0000 7fd4dd600700 1 --2- 172.21.15.12:0/756151007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4d8074e20 0x7fd4d8112f60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40099c0 tx=0x7fd4c4009cd0 comp rx=0 tx=0).stop 2024-03-17T23:56:38.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.881+0000 7fd4dd600700 1 -- 172.21.15.12:0/756151007 shutdown_connections 2024-03-17T23:56:38.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.881+0000 7fd4dd600700 1 --2- 172.21.15.12:0/756151007 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4d81134a0 0x7fd4d8115890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:38.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.881+0000 7fd4dd600700 1 --2- 172.21.15.12:0/756151007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4d8074500 0x7fd4d80748e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:38.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.881+0000 7fd4dd600700 1 --2- 172.21.15.12:0/756151007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4d8074e20 0x7fd4d8112f60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:38.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.881+0000 7fd4dd600700 1 -- 172.21.15.12:0/756151007 >> 172.21.15.12:0/756151007 conn(0x7fd4d8079430 msgr2=0x7fd4d8079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:38.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.881+0000 7fd4dd600700 1 -- 172.21.15.12:0/756151007 shutdown_connections 2024-03-17T23:56:38.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.881+0000 7fd4dd600700 1 -- 172.21.15.12:0/756151007 wait complete. 2024-03-17T23:56:38.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.882+0000 7fd4dd600700 1 Processor -- start 2024-03-17T23:56:38.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.883+0000 7fd4dd600700 1 -- start start 2024-03-17T23:56:38.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.883+0000 7fd4dd600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4d8074500 0x7fd4d819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:38.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.883+0000 7fd4dd600700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4d8074e20 0x7fd4d819f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:38.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.883+0000 7fd4dd600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4d81134a0 0x7fd4d81a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:38.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.883+0000 7fd4dd600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d8118ce0 con 0x7fd4d81134a0 2024-03-17T23:56:38.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.883+0000 7fd4dd600700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d8118b60 con 0x7fd4d8074e20 2024-03-17T23:56:38.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.883+0000 7fd4dd600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d8118e60 con 0x7fd4d8074500 2024-03-17T23:56:38.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.883+0000 7fd4d6000700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4d8074e20 0x7fd4d819f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:38.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.883+0000 7fd4d6a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4d8074500 0x7fd4d819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:38.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.883+0000 7fd4d6000700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4d8074e20 0x7fd4d819f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:41146/0 (socket says 172.21.15.12:41146) 2024-03-17T23:56:38.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.883+0000 7fd4d6000700 1 -- 172.21.15.12:0/2447800102 learned_addr learned my addr 172.21.15.12:0/2447800102 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:38.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.883+0000 7fd4d7400700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4d81134a0 0x7fd4d81a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:38.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.884+0000 7fd4d6a00700 1 -- 172.21.15.12:0/2447800102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4d8074e20 msgr2=0x7fd4d819f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:38.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.884+0000 7fd4d6a00700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4d8074e20 0x7fd4d819f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:38.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.884+0000 7fd4d6a00700 1 -- 172.21.15.12:0/2447800102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4d81134a0 msgr2=0x7fd4d81a39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:38.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.884+0000 7fd4d6a00700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4d81134a0 0x7fd4d81a39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:38.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.884+0000 7fd4d6a00700 1 -- 172.21.15.12:0/2447800102 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d81a40e0 con 0x7fd4d8074500 2024-03-17T23:56:38.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.884+0000 7fd4d7400700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4d81134a0 0x7fd4d81a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:38.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.884+0000 7fd4d6a00700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4d8074500 0x7fd4d819f0d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd4c800c970 tx=0x7fd4c800cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:38.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.885+0000 7fd4cf400700 1 -- 172.21.15.12:0/2447800102 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8014070 con 0x7fd4d8074500 2024-03-17T23:56:38.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.885+0000 7fd4cf400700 1 -- 172.21.15.12:0/2447800102 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4c8004440 con 0x7fd4d8074500 2024-03-17T23:56:38.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.885+0000 7fd4dd600700 1 -- 172.21.15.12:0/2447800102 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d81a43d0 con 0x7fd4d8074500 2024-03-17T23:56:38.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.885+0000 7fd4cf400700 1 -- 172.21.15.12:0/2447800102 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8007b60 con 0x7fd4d8074500 2024-03-17T23:56:38.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.885+0000 7fd4dd600700 1 -- 172.21.15.12:0/2447800102 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d81abc50 con 0x7fd4d8074500 2024-03-17T23:56:38.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.886+0000 7fd4dd600700 1 -- 172.21.15.12:0/2447800102 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4d8068f90 con 0x7fd4d8074500 2024-03-17T23:56:38.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.888+0000 7fd4cf400700 1 -- 172.21.15.12:0/2447800102 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd4c8017070 con 0x7fd4d8074500 2024-03-17T23:56:38.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.889+0000 7fd4cf400700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd4bc0a29a0 0x7fd4bc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:38.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.890+0000 7fd4cf400700 1 -- 172.21.15.12:0/2447800102 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fd4c80ceb30 con 0x7fd4d8074500 2024-03-17T23:56:38.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.890+0000 7fd4d6000700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd4bc0a29a0 0x7fd4bc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:38.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.890+0000 7fd4d6000700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd4bc0a29a0 0x7fd4bc0a4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8115380 tx=0x7fd4c4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:38.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:38.898+0000 7fd4cf400700 1 -- 172.21.15.12:0/2447800102 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd4c801e020 con 0x7fd4d8074500 2024-03-17T23:56:39.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.138+0000 7fd4dd600700 1 -- 172.21.15.12:0/2447800102 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd4d81abee0 con 0x7fd4d8074500 2024-03-17T23:56:39.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.138+0000 7fd4cf400700 1 -- 172.21.15.12:0/2447800102 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd4c809b6b0 con 0x7fd4d8074500 2024-03-17T23:56:39.139 INFO:teuthology.orchestra.run.smithi012.stdout:55834574939 2024-03-17T23:56:39.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.141+0000 7fd4dd600700 1 -- 172.21.15.12:0/2447800102 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd4bc0a29a0 msgr2=0x7fd4bc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:39.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.141+0000 7fd4dd600700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd4bc0a29a0 0x7fd4bc0a4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8115380 tx=0x7fd4c4005fb0 comp rx=0 tx=0).stop 2024-03-17T23:56:39.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.141+0000 7fd4dd600700 1 -- 172.21.15.12:0/2447800102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4d8074500 msgr2=0x7fd4d819f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:39.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.141+0000 7fd4dd600700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4d8074500 0x7fd4d819f0d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd4c800c970 tx=0x7fd4c800cc80 comp rx=0 tx=0).stop 2024-03-17T23:56:39.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.142+0000 7fd4dd600700 1 -- 172.21.15.12:0/2447800102 shutdown_connections 2024-03-17T23:56:39.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.142+0000 7fd4dd600700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd4bc0a29a0 0x7fd4bc0a4e60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:39.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.142+0000 7fd4dd600700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4d81134a0 0x7fd4d81a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:39.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.142+0000 7fd4dd600700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4d8074500 0x7fd4d819f0d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:39.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.142+0000 7fd4dd600700 1 --2- 172.21.15.12:0/2447800102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4d8074e20 0x7fd4d819f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:39.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.142+0000 7fd4dd600700 1 -- 172.21.15.12:0/2447800102 >> 172.21.15.12:0/2447800102 conn(0x7fd4d8079430 msgr2=0x7fd4d8112260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:39.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.142+0000 7fd4dd600700 1 -- 172.21.15.12:0/2447800102 shutdown_connections 2024-03-17T23:56:39.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:39.142+0000 7fd4dd600700 1 -- 172.21.15.12:0/2447800102 wait complete. 2024-03-17T23:56:39.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:39 smithi012 ceph-mon[31541]: pgmap v374: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:39.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:39 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2447800102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-17T23:56:39.828 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182473 got 107374182479 for osd.3 2024-03-17T23:56:39.828 DEBUG:teuthology.parallel:result is None 2024-03-17T23:56:39.893 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:39 smithi191 ceph-mon[36708]: pgmap v374: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:39.893 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:39 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2447800102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-17T23:56:39.940 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411407 got 77309411413 for osd.2 2024-03-17T23:56:39.940 DEBUG:teuthology.parallel:result is None 2024-03-17T23:56:40.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:39 smithi079 ceph-mon[36241]: pgmap v374: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:40.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:39 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2447800102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-17T23:56:40.517 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:41.122 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070509 got 253403070514 for osd.8 2024-03-17T23:56:41.122 DEBUG:teuthology.parallel:result is None 2024-03-17T23:56:41.711 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574932 got 55834574939 for osd.1 2024-03-17T23:56:41.711 DEBUG:teuthology.parallel:result is None 2024-03-17T23:56:41.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:41 smithi012 ceph-mon[31541]: pgmap v375: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.957+0000 7fa92d600700 1 -- 172.21.15.12:0/73032876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa928069f80 msgr2=0x7fa92806a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.957+0000 7fa92d600700 1 --2- 172.21.15.12:0/73032876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa928069f80 0x7fa92806a400 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa914009990 tx=0x7fa914009ca0 comp rx=0 tx=0).stop 2024-03-17T23:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.957+0000 7fa92d600700 1 -- 172.21.15.12:0/73032876 shutdown_connections 2024-03-17T23:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.957+0000 7fa92d600700 1 --2- 172.21.15.12:0/73032876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa92806a940 0x7fa92810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.957+0000 7fa92d600700 1 --2- 172.21.15.12:0/73032876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa928108300 0x7fa9281086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.957+0000 7fa92d600700 1 --2- 172.21.15.12:0/73032876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa928069f80 0x7fa92806a400 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.957+0000 7fa92d600700 1 -- 172.21.15.12:0/73032876 >> 172.21.15.12:0/73032876 conn(0x7fa928077f80 msgr2=0x7fa928078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.957+0000 7fa92d600700 1 -- 172.21.15.12:0/73032876 shutdown_connections 2024-03-17T23:56:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.957+0000 7fa92d600700 1 -- 172.21.15.12:0/73032876 wait complete. 2024-03-17T23:56:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.958+0000 7fa92d600700 1 Processor -- start 2024-03-17T23:56:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.959+0000 7fa92d600700 1 -- start start 2024-03-17T23:56:41.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.959+0000 7fa92d600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa928069f80 0x7fa92819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:41.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.959+0000 7fa92d600700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa92806a940 0x7fa92819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:41.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.959+0000 7fa92d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa928108300 0x7fa928198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:41.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.959+0000 7fa92d600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa9281125d0 con 0x7fa928108300 2024-03-17T23:56:41.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.959+0000 7fa92d600700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa928112450 con 0x7fa92806a940 2024-03-17T23:56:41.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.959+0000 7fa92d600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa928112750 con 0x7fa928069f80 2024-03-17T23:56:41.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.959+0000 7fa926a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa928069f80 0x7fa92819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:41.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.959+0000 7fa926a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa928069f80 0x7fa92819e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:36076/0 (socket says 172.21.15.12:36076) 2024-03-17T23:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.959+0000 7fa926a00700 1 -- 172.21.15.12:0/2196410411 learned_addr learned my addr 172.21.15.12:0/2196410411 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.960+0000 7fa926000700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa92806a940 0x7fa92819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.960+0000 7fa926a00700 1 -- 172.21.15.12:0/2196410411 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa92806a940 msgr2=0x7fa92819ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.960+0000 7fa926a00700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa92806a940 0x7fa92819ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.960+0000 7fa927400700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa928108300 0x7fa928198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.960+0000 7fa926a00700 1 -- 172.21.15.12:0/2196410411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa928108300 msgr2=0x7fa928198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.960+0000 7fa926a00700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa928108300 0x7fa928198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.960+0000 7fa926a00700 1 -- 172.21.15.12:0/2196410411 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa928199050 con 0x7fa928069f80 2024-03-17T23:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.960+0000 7fa927400700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa928108300 0x7fa928198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.960+0000 7fa926a00700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa928069f80 0x7fa92819e6a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa91800e9a0 tx=0x7fa91800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.961+0000 7fa91f400700 1 -- 172.21.15.12:0/2196410411 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91800cbd0 con 0x7fa928069f80 2024-03-17T23:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.961+0000 7fa91f400700 1 -- 172.21.15.12:0/2196410411 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa91800cd70 con 0x7fa928069f80 2024-03-17T23:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.961+0000 7fa91f400700 1 -- 172.21.15.12:0/2196410411 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa918018750 con 0x7fa928069f80 2024-03-17T23:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.961+0000 7fa92d600700 1 -- 172.21.15.12:0/2196410411 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa928199340 con 0x7fa928069f80 2024-03-17T23:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.961+0000 7fa92d600700 1 -- 172.21.15.12:0/2196410411 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9281a5500 con 0x7fa928069f80 2024-03-17T23:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.962+0000 7fa92d600700 1 -- 172.21.15.12:0/2196410411 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa928103990 con 0x7fa928069f80 2024-03-17T23:56:41.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.964+0000 7fa91f400700 1 -- 172.21.15.12:0/2196410411 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa918010430 con 0x7fa928069f80 2024-03-17T23:56:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.966+0000 7fa91f400700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa90c0a29a0 0x7fa90c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.966+0000 7fa91f400700 1 -- 172.21.15.12:0/2196410411 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fa9180cf130 con 0x7fa928069f80 2024-03-17T23:56:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.966+0000 7fa926000700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa90c0a29a0 0x7fa90c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:41.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.969+0000 7fa926000700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa90c0a29a0 0x7fa90c0a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa9280785f0 tx=0x7fa91400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:41.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:41.969+0000 7fa91f400700 1 -- 172.21.15.12:0/2196410411 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa91809bcb0 con 0x7fa928069f80 2024-03-17T23:56:42.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:41 smithi079 ceph-mon[36241]: pgmap v375: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:42.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:41 smithi191 ceph-mon[36708]: pgmap v375: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:42.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.203+0000 7fa92d600700 1 -- 172.21.15.12:0/2196410411 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fa92802db70 con 0x7fa928069f80 2024-03-17T23:56:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.203+0000 7fa91f400700 1 -- 172.21.15.12:0/2196410411 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fa91809b5b0 con 0x7fa928069f80 2024-03-17T23:56:42.204 INFO:teuthology.orchestra.run.smithi012.stdout:309237645351 2024-03-17T23:56:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.206+0000 7fa92d600700 1 -- 172.21.15.12:0/2196410411 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa90c0a29a0 msgr2=0x7fa90c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.206+0000 7fa92d600700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa90c0a29a0 0x7fa90c0a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa9280785f0 tx=0x7fa91400b260 comp rx=0 tx=0).stop 2024-03-17T23:56:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.206+0000 7fa92d600700 1 -- 172.21.15.12:0/2196410411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa928069f80 msgr2=0x7fa92819e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.206+0000 7fa92d600700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa928069f80 0x7fa92819e6a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa91800e9a0 tx=0x7fa91800ecb0 comp rx=0 tx=0).stop 2024-03-17T23:56:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.206+0000 7fa92d600700 1 -- 172.21.15.12:0/2196410411 shutdown_connections 2024-03-17T23:56:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.206+0000 7fa92d600700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa90c0a29a0 0x7fa90c0a4e60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.206+0000 7fa92d600700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa928108300 0x7fa928198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.206+0000 7fa92d600700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa928069f80 0x7fa92819e6a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.206+0000 7fa92d600700 1 --2- 172.21.15.12:0/2196410411 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa92806a940 0x7fa92819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.206+0000 7fa92d600700 1 -- 172.21.15.12:0/2196410411 >> 172.21.15.12:0/2196410411 conn(0x7fa928077f80 msgr2=0x7fa928100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.206+0000 7fa92d600700 1 -- 172.21.15.12:0/2196410411 shutdown_connections 2024-03-17T23:56:42.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:42.206+0000 7fa92d600700 1 -- 172.21.15.12:0/2196410411 wait complete. 2024-03-17T23:56:42.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:42 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2196410411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-17T23:56:43.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:42 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2196410411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-17T23:56:43.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:42 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2196410411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-17T23:56:43.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:43 smithi012 ceph-mon[31541]: pgmap v376: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:56:43.935 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645343 got 309237645351 for osd.10 2024-03-17T23:56:43.936 DEBUG:teuthology.parallel:result is None 2024-03-17T23:56:44.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:43 smithi079 ceph-mon[36241]: pgmap v376: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:56:44.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:43 smithi191 ceph-mon[36708]: pgmap v376: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:56:45.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.805+0000 7f3103000700 1 -- 172.21.15.12:0/617253098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30fc074700 msgr2=0x7f30fc074ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:45.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.805+0000 7f3103000700 1 --2- 172.21.15.12:0/617253098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30fc074700 0x7f30fc074ae0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009990 tx=0x7f30ec009ca0 comp rx=0 tx=0).stop 2024-03-17T23:56:45.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.806+0000 7f3103000700 1 -- 172.21.15.12:0/617253098 shutdown_connections 2024-03-17T23:56:45.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.806+0000 7f3103000700 1 --2- 172.21.15.12:0/617253098 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30fc10eb30 0x7f30fc110fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:45.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.806+0000 7f3103000700 1 --2- 172.21.15.12:0/617253098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30fc074700 0x7f30fc074ae0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:45.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.806+0000 7f3103000700 1 --2- 172.21.15.12:0/617253098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30fc075020 0x7f30fc10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:45.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.806+0000 7f3103000700 1 -- 172.21.15.12:0/617253098 >> 172.21.15.12:0/617253098 conn(0x7f30fc0fdda0 msgr2=0x7f30fc1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:45.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.806+0000 7f3103000700 1 -- 172.21.15.12:0/617253098 shutdown_connections 2024-03-17T23:56:45.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.806+0000 7f3103000700 1 -- 172.21.15.12:0/617253098 wait complete. 2024-03-17T23:56:45.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.808+0000 7f3103000700 1 Processor -- start 2024-03-17T23:56:45.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.808+0000 7f3103000700 1 -- start start 2024-03-17T23:56:45.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.809+0000 7f3103000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30fc074700 0x7f30fc1a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:45.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.809+0000 7f3103000700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30fc075020 0x7f30fc1a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:45.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.809+0000 7f3103000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30fc10eb30 0x7f30fc19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:45.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.809+0000 7f3103000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc114860 con 0x7f30fc10eb30 2024-03-17T23:56:45.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.809+0000 7f3103000700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc1146e0 con 0x7f30fc075020 2024-03-17T23:56:45.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.809+0000 7f3103000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc1149e0 con 0x7f30fc074700 2024-03-17T23:56:45.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.809+0000 7f30fbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30fc074700 0x7f30fc1a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:45.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.809+0000 7f30fbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30fc074700 0x7f30fc1a0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:58228/0 (socket says 172.21.15.12:58228) 2024-03-17T23:56:45.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.809+0000 7f30fbe00700 1 -- 172.21.15.12:0/3079747537 learned_addr learned my addr 172.21.15.12:0/3079747537 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:45.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.809+0000 7f30fb400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30fc075020 0x7f30fc1a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:45.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.810+0000 7f30fb400700 1 -- 172.21.15.12:0/3079747537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30fc074700 msgr2=0x7f30fc1a0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:45.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.810+0000 7f30fb400700 1 --2- 172.21.15.12:0/3079747537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30fc074700 0x7f30fc1a0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:45.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.810+0000 7f30fb400700 1 -- 172.21.15.12:0/3079747537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30fc10eb30 msgr2=0x7f30fc19ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:45.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.810+0000 7f30fb400700 1 --2- 172.21.15.12:0/3079747537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30fc10eb30 0x7f30fc19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:45.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.810+0000 7f30fb400700 1 -- 172.21.15.12:0/3079747537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30fc19b460 con 0x7f30fc075020 2024-03-17T23:56:45.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.810+0000 7f30fb400700 1 --2- 172.21.15.12:0/3079747537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30fc075020 0x7f30fc1a0fa0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f30e800e9f0 tx=0x7f30e800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:45.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.810+0000 7f30f8c00700 1 -- 172.21.15.12:0/3079747537 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e800c9c0 con 0x7f30fc075020 2024-03-17T23:56:45.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.810+0000 7f30f8c00700 1 -- 172.21.15.12:0/3079747537 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30e800cb60 con 0x7f30fc075020 2024-03-17T23:56:45.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.810+0000 7f30f8c00700 1 -- 172.21.15.12:0/3079747537 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e8018660 con 0x7f30fc075020 2024-03-17T23:56:45.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.810+0000 7f3103000700 1 -- 172.21.15.12:0/3079747537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30fc19b750 con 0x7f30fc075020 2024-03-17T23:56:45.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.811+0000 7f3103000700 1 -- 172.21.15.12:0/3079747537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30fc1a79e0 con 0x7f30fc075020 2024-03-17T23:56:45.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.812+0000 7f3103000700 1 -- 172.21.15.12:0/3079747537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30fc075db0 con 0x7f30fc075020 2024-03-17T23:56:45.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.813+0000 7f30f8c00700 1 -- 172.21.15.12:0/3079747537 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f30e8018ad0 con 0x7f30fc075020 2024-03-17T23:56:45.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.815+0000 7f30f8c00700 1 --2- 172.21.15.12:0/3079747537 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f30e00a29a0 0x7f30e00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:45.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.815+0000 7f30f8c00700 1 -- 172.21.15.12:0/3079747537 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f30e80cf2b0 con 0x7f30fc075020 2024-03-17T23:56:45.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.815+0000 7f30fbe00700 1 --2- 172.21.15.12:0/3079747537 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f30e00a29a0 0x7f30e00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:45.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.816+0000 7f30fbe00700 1 --2- 172.21.15.12:0/3079747537 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f30e00a29a0 0x7f30e00a4e60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f30ec006010 tx=0x7f30ec00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:45.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:45.819+0000 7f30f8c00700 1 -- 172.21.15.12:0/3079747537 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30e809be30 con 0x7f30fc075020 2024-03-17T23:56:45.892 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:45 smithi012 ceph-mon[31541]: pgmap v377: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:46.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:45 smithi079 ceph-mon[36241]: pgmap v377: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:45 smithi191 ceph-mon[36708]: pgmap v377: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:46.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.093+0000 7f3103000700 1 -- 172.21.15.12:0/3079747537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f30fc19bff0 con 0x7f30fc075020 2024-03-17T23:56:46.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.094+0000 7f30f8c00700 1 -- 172.21.15.12:0/3079747537 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f30e809b730 con 0x7f30fc075020 2024-03-17T23:56:46.094 INFO:teuthology.orchestra.run.smithi012.stdout:34359738464 2024-03-17T23:56:46.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.097+0000 7f3103000700 1 -- 172.21.15.12:0/3079747537 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f30e00a29a0 msgr2=0x7f30e00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:46.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.097+0000 7f3103000700 1 --2- 172.21.15.12:0/3079747537 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f30e00a29a0 0x7f30e00a4e60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f30ec006010 tx=0x7f30ec00b260 comp rx=0 tx=0).stop 2024-03-17T23:56:46.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.097+0000 7f3103000700 1 -- 172.21.15.12:0/3079747537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30fc075020 msgr2=0x7f30fc1a0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:46.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.097+0000 7f3103000700 1 --2- 172.21.15.12:0/3079747537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30fc075020 0x7f30fc1a0fa0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f30e800e9f0 tx=0x7f30e800ed00 comp rx=0 tx=0).stop 2024-03-17T23:56:46.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.097+0000 7f3103000700 1 -- 172.21.15.12:0/3079747537 shutdown_connections 2024-03-17T23:56:46.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.097+0000 7f3103000700 1 --2- 172.21.15.12:0/3079747537 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f30e00a29a0 0x7f30e00a4e60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.097+0000 7f3103000700 1 --2- 172.21.15.12:0/3079747537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30fc10eb30 0x7f30fc19ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.097+0000 7f3103000700 1 --2- 172.21.15.12:0/3079747537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30fc074700 0x7f30fc1a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.097+0000 7f3103000700 1 --2- 172.21.15.12:0/3079747537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30fc075020 0x7f30fc1a0fa0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.097+0000 7f3103000700 1 -- 172.21.15.12:0/3079747537 >> 172.21.15.12:0/3079747537 conn(0x7f30fc0fdda0 msgr2=0x7f30fc0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:46.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.097+0000 7f3103000700 1 -- 172.21.15.12:0/3079747537 shutdown_connections 2024-03-17T23:56:46.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.097+0000 7f3103000700 1 -- 172.21.15.12:0/3079747537 wait complete. 2024-03-17T23:56:46.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.672+0000 7f3f08600700 1 -- 172.21.15.12:0/3509994798 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f00069f80 msgr2=0x7f3f0006a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:46.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.672+0000 7f3f08600700 1 --2- 172.21.15.12:0/3509994798 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f00069f80 0x7f3f0006a3e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009990 tx=0x7f3eec009ca0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.672+0000 7f3f08600700 1 -- 172.21.15.12:0/3509994798 shutdown_connections 2024-03-17T23:56:46.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.672+0000 7f3f08600700 1 --2- 172.21.15.12:0/3509994798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f0006a920 0x7f3f0010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.672+0000 7f3f08600700 1 --2- 172.21.15.12:0/3509994798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f00108320 0x7f3f00108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.672+0000 7f3f08600700 1 --2- 172.21.15.12:0/3509994798 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f00069f80 0x7f3f0006a3e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.672+0000 7f3f08600700 1 -- 172.21.15.12:0/3509994798 >> 172.21.15.12:0/3509994798 conn(0x7f3f00078100 msgr2=0x7f3f00078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:46.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.673+0000 7f3f08600700 1 -- 172.21.15.12:0/3509994798 shutdown_connections 2024-03-17T23:56:46.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.673+0000 7f3f08600700 1 -- 172.21.15.12:0/3509994798 wait complete. 2024-03-17T23:56:46.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.674+0000 7f3f08600700 1 Processor -- start 2024-03-17T23:56:46.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.674+0000 7f3f08600700 1 -- start start 2024-03-17T23:56:46.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.674+0000 7f3f08600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f00069f80 0x7f3f001a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:46.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.674+0000 7f3f08600700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f0006a920 0x7f3f001a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:46.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.674+0000 7f3f08600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f00108320 0x7f3f0019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:46.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.674+0000 7f3f08600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3f001125d0 con 0x7f3f00108320 2024-03-17T23:56:46.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.674+0000 7f3f08600700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3f00112450 con 0x7f3f0006a920 2024-03-17T23:56:46.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.674+0000 7f3f08600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3f00112750 con 0x7f3f00069f80 2024-03-17T23:56:46.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.675+0000 7f3f05200700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f0006a920 0x7f3f001a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:46.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.675+0000 7f3f05c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f00069f80 0x7f3f001a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:46.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.675+0000 7f3f05200700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f0006a920 0x7f3f001a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:40928/0 (socket says 172.21.15.12:40928) 2024-03-17T23:56:46.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.675+0000 7f3f05200700 1 -- 172.21.15.12:0/1480963451 learned_addr learned my addr 172.21.15.12:0/1480963451 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:46.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.675+0000 7f3f06600700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f00108320 0x7f3f0019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:46.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.675+0000 7f3f05200700 1 -- 172.21.15.12:0/1480963451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f00069f80 msgr2=0x7f3f001a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:46.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.675+0000 7f3f05200700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f00069f80 0x7f3f001a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.675+0000 7f3f05200700 1 -- 172.21.15.12:0/1480963451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f00108320 msgr2=0x7f3f0019ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:46.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.675+0000 7f3f05200700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f00108320 0x7f3f0019ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.675+0000 7f3f05200700 1 -- 172.21.15.12:0/1480963451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0019b260 con 0x7f3f0006a920 2024-03-17T23:56:46.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.675+0000 7f3f05c00700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f00069f80 0x7f3f001a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:46.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.675+0000 7f3f05200700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f0006a920 0x7f3f001a0e80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3eec000c00 tx=0x7f3eec0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:46.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.676+0000 7f3efe000700 1 -- 172.21.15.12:0/1480963451 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec027070 con 0x7f3f0006a920 2024-03-17T23:56:46.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.676+0000 7f3efe000700 1 -- 172.21.15.12:0/1480963451 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eec00be70 con 0x7f3f0006a920 2024-03-17T23:56:46.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.676+0000 7f3f08600700 1 -- 172.21.15.12:0/1480963451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f0019b4f0 con 0x7f3f0006a920 2024-03-17T23:56:46.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.676+0000 7f3efe000700 1 -- 172.21.15.12:0/1480963451 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec017650 con 0x7f3f0006a920 2024-03-17T23:56:46.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.676+0000 7f3f08600700 1 -- 172.21.15.12:0/1480963451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f001a7810 con 0x7f3f0006a920 2024-03-17T23:56:46.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.677+0000 7f3f08600700 1 -- 172.21.15.12:0/1480963451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f00068f90 con 0x7f3f0006a920 2024-03-17T23:56:46.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.679+0000 7f3efe000700 1 -- 172.21.15.12:0/1480963451 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3eec02a070 con 0x7f3f0006a920 2024-03-17T23:56:46.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.680+0000 7f3efe000700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ee80a29a0 0x7f3ee80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:46.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.680+0000 7f3efe000700 1 -- 172.21.15.12:0/1480963451 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f3eec0cff10 con 0x7f3f0006a920 2024-03-17T23:56:46.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.680+0000 7f3f05c00700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ee80a29a0 0x7f3ee80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:46.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.681+0000 7f3f05c00700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ee80a29a0 0x7f3ee80a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4009fd0 tx=0x7f3ef4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:46.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.684+0000 7f3efe000700 1 -- 172.21.15.12:0/1480963451 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3eec05e740 con 0x7f3f0006a920 2024-03-17T23:56:46.735 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:46 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3079747537' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-17T23:56:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.943+0000 7f3f08600700 1 -- 172.21.15.12:0/1480963451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3f001a79e0 con 0x7f3f0006a920 2024-03-17T23:56:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.943+0000 7f3efe000700 1 -- 172.21.15.12:0/1480963451 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3eec013070 con 0x7f3f0006a920 2024-03-17T23:56:46.944 INFO:teuthology.orchestra.run.smithi012.stdout:163208757318 2024-03-17T23:56:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.946+0000 7f3f08600700 1 -- 172.21.15.12:0/1480963451 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ee80a29a0 msgr2=0x7f3ee80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.946+0000 7f3f08600700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ee80a29a0 0x7f3ee80a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4009fd0 tx=0x7f3ef4009260 comp rx=0 tx=0).stop 2024-03-17T23:56:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.946+0000 7f3f08600700 1 -- 172.21.15.12:0/1480963451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f0006a920 msgr2=0x7f3f001a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.946+0000 7f3f08600700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f0006a920 0x7f3f001a0e80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3eec000c00 tx=0x7f3eec0057c0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.946+0000 7f3f08600700 1 -- 172.21.15.12:0/1480963451 shutdown_connections 2024-03-17T23:56:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.946+0000 7f3f08600700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ee80a29a0 0x7f3ee80a4e60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.946+0000 7f3f08600700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f00108320 0x7f3f0019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.946+0000 7f3f08600700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f00069f80 0x7f3f001a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.946+0000 7f3f08600700 1 --2- 172.21.15.12:0/1480963451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f0006a920 0x7f3f001a0e80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.946+0000 7f3f08600700 1 -- 172.21.15.12:0/1480963451 >> 172.21.15.12:0/1480963451 conn(0x7f3f00078100 msgr2=0x7f3f0010d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.947+0000 7f3f08600700 1 -- 172.21.15.12:0/1480963451 shutdown_connections 2024-03-17T23:56:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:46.947+0000 7f3f08600700 1 -- 172.21.15.12:0/1480963451 wait complete. 2024-03-17T23:56:47.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:46 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3079747537' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-17T23:56:47.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:46 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3079747537' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-17T23:56:47.454 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738459 got 34359738464 for osd.0 2024-03-17T23:56:47.454 DEBUG:teuthology.parallel:result is None 2024-03-17T23:56:47.947 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:47 smithi079 ceph-mon[36241]: pgmap v378: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:56:47.947 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:47 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:47.947 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:47 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:47.947 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:47 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1480963451' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-17T23:56:48.006 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:47 smithi012 ceph-mon[31541]: pgmap v378: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:56:48.006 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:47 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:48.006 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:47 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:48.006 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:47 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1480963451' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-17T23:56:48.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:47 smithi191 ceph-mon[36708]: pgmap v378: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:56:48.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:47 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:48.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:47 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:48.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:47 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1480963451' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-17T23:56:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.190+0000 7fcac3400700 1 -- 172.21.15.12:0/2738975783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcabc06a4c0 msgr2=0x7fcabc06a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.190+0000 7fcac3400700 1 --2- 172.21.15.12:0/2738975783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcabc06a4c0 0x7fcabc06a920 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcaa80098f0 tx=0x7fcaa8009c00 comp rx=0 tx=0).stop 2024-03-17T23:56:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.190+0000 7fcac3400700 1 -- 172.21.15.12:0/2738975783 shutdown_connections 2024-03-17T23:56:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.190+0000 7fcac3400700 1 --2- 172.21.15.12:0/2738975783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcabc104cd0 0x7fcabc105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.190+0000 7fcac3400700 1 --2- 172.21.15.12:0/2738975783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcabc1071c0 0x7fcabc069f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.190+0000 7fcac3400700 1 --2- 172.21.15.12:0/2738975783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcabc06a4c0 0x7fcabc06a920 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.190+0000 7fcac3400700 1 -- 172.21.15.12:0/2738975783 >> 172.21.15.12:0/2738975783 conn(0x7fcabc078100 msgr2=0x7fcabc078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.190+0000 7fcac3400700 1 -- 172.21.15.12:0/2738975783 shutdown_connections 2024-03-17T23:56:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.191+0000 7fcac3400700 1 -- 172.21.15.12:0/2738975783 wait complete. 2024-03-17T23:56:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.192+0000 7fcac3400700 1 Processor -- start 2024-03-17T23:56:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.192+0000 7fcac3400700 1 -- start start 2024-03-17T23:56:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.192+0000 7fcac3400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcabc06a4c0 0x7fcabc19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.192+0000 7fcac3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcabc104cd0 0x7fcabc19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.192+0000 7fcac3400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcabc1071c0 0x7fcabc1a3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.192+0000 7fcac3400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcabc118bc0 con 0x7fcabc104cd0 2024-03-17T23:56:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.192+0000 7fcac3400700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcabc118a40 con 0x7fcabc1071c0 2024-03-17T23:56:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.192+0000 7fcac3400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcabc118d40 con 0x7fcabc06a4c0 2024-03-17T23:56:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.192+0000 7fcac0a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcabc06a4c0 0x7fcabc19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.193+0000 7fcac1400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcabc1071c0 0x7fcabc1a3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.193+0000 7fcabbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcabc104cd0 0x7fcabc19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.193+0000 7fcac0a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcabc06a4c0 0x7fcabc19f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:58272/0 (socket says 172.21.15.12:58272) 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.193+0000 7fcac0a00700 1 -- 172.21.15.12:0/3621623527 learned_addr learned my addr 172.21.15.12:0/3621623527 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.193+0000 7fcac1400700 1 -- 172.21.15.12:0/3621623527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcabc06a4c0 msgr2=0x7fcabc19f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.193+0000 7fcac1400700 1 --2- 172.21.15.12:0/3621623527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcabc06a4c0 0x7fcabc19f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.193+0000 7fcac1400700 1 -- 172.21.15.12:0/3621623527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcabc104cd0 msgr2=0x7fcabc19f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.193+0000 7fcac1400700 1 --2- 172.21.15.12:0/3621623527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcabc104cd0 0x7fcabc19f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.193+0000 7fcac1400700 1 -- 172.21.15.12:0/3621623527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcabc1a4140 con 0x7fcabc1071c0 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.193+0000 7fcac0a00700 1 --2- 172.21.15.12:0/3621623527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcabc06a4c0 0x7fcabc19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.193+0000 7fcac1400700 1 --2- 172.21.15.12:0/3621623527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcabc1071c0 0x7fcabc1a3a00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcab400eb10 tx=0x7fcab400ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.193+0000 7fcab9600700 1 -- 172.21.15.12:0/3621623527 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab400cb70 con 0x7fcabc1071c0 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.194+0000 7fcab9600700 1 -- 172.21.15.12:0/3621623527 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcab4004510 con 0x7fcabc1071c0 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.194+0000 7fcab9600700 1 -- 172.21.15.12:0/3621623527 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab4010430 con 0x7fcabc1071c0 2024-03-17T23:56:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.194+0000 7fcac3400700 1 -- 172.21.15.12:0/3621623527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcabc1a4430 con 0x7fcabc1071c0 2024-03-17T23:56:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.194+0000 7fcac3400700 1 -- 172.21.15.12:0/3621623527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcabc1abcb0 con 0x7fcabc1071c0 2024-03-17T23:56:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.195+0000 7fcac3400700 1 -- 172.21.15.12:0/3621623527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcabc068f90 con 0x7fcabc1071c0 2024-03-17T23:56:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.198+0000 7fcab9600700 1 -- 172.21.15.12:0/3621623527 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fcab4003680 con 0x7fcabc1071c0 2024-03-17T23:56:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.199+0000 7fcab9600700 1 --2- 172.21.15.12:0/3621623527 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcaa40a29a0 0x7fcaa40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.199+0000 7fcab9600700 1 -- 172.21.15.12:0/3621623527 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fcab40cf180 con 0x7fcabc1071c0 2024-03-17T23:56:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.199+0000 7fcac0a00700 1 --2- 172.21.15.12:0/3621623527 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcaa40a29a0 0x7fcaa40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.200+0000 7fcac0a00700 1 --2- 172.21.15.12:0/3621623527 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcaa40a29a0 0x7fcaa40a4e60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009c10 tx=0x7fcab000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.203+0000 7fcab9600700 1 -- 172.21.15.12:0/3621623527 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcab409bd00 con 0x7fcabc1071c0 2024-03-17T23:56:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.437+0000 7fcac3400700 1 -- 172.21.15.12:0/3621623527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fcabc1abe80 con 0x7fcabc1071c0 2024-03-17T23:56:48.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.438+0000 7fcab9600700 1 -- 172.21.15.12:0/3621623527 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fcab409b600 con 0x7fcabc1071c0 2024-03-17T23:56:48.438 INFO:teuthology.orchestra.run.smithi012.stdout:343597383716 2024-03-17T23:56:48.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.442+0000 7fcac3400700 1 -- 172.21.15.12:0/3621623527 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcaa40a29a0 msgr2=0x7fcaa40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:48.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.442+0000 7fcac3400700 1 --2- 172.21.15.12:0/3621623527 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcaa40a29a0 0x7fcaa40a4e60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009c10 tx=0x7fcab000b040 comp rx=0 tx=0).stop 2024-03-17T23:56:48.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.442+0000 7fcac3400700 1 -- 172.21.15.12:0/3621623527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcabc1071c0 msgr2=0x7fcabc1a3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:48.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.442+0000 7fcac3400700 1 --2- 172.21.15.12:0/3621623527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcabc1071c0 0x7fcabc1a3a00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcab400eb10 tx=0x7fcab400ee20 comp rx=0 tx=0).stop 2024-03-17T23:56:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.442+0000 7fcac3400700 1 -- 172.21.15.12:0/3621623527 shutdown_connections 2024-03-17T23:56:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.442+0000 7fcac3400700 1 --2- 172.21.15.12:0/3621623527 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcaa40a29a0 0x7fcaa40a4e60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.442+0000 7fcac3400700 1 --2- 172.21.15.12:0/3621623527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcabc1071c0 0x7fcabc1a3a00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.442+0000 7fcac3400700 1 --2- 172.21.15.12:0/3621623527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcabc06a4c0 0x7fcabc19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.442+0000 7fcac3400700 1 --2- 172.21.15.12:0/3621623527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcabc104cd0 0x7fcabc19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.442+0000 7fcac3400700 1 -- 172.21.15.12:0/3621623527 >> 172.21.15.12:0/3621623527 conn(0x7fcabc078100 msgr2=0x7fcabc101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.442+0000 7fcac3400700 1 -- 172.21.15.12:0/3621623527 shutdown_connections 2024-03-17T23:56:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:48.442+0000 7fcac3400700 1 -- 172.21.15.12:0/3621623527 wait complete. 2024-03-17T23:56:48.625 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757312 got 163208757318 for osd.5 2024-03-17T23:56:48.625 DEBUG:teuthology.parallel:result is None 2024-03-17T23:56:48.986 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:56:48.986 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:56:48.986 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:48.986 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:48 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3621623527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-17T23:56:49.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:56:49.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:56:49.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:49.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:48 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3621623527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-17T23:56:49.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:56:49.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:56:49.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:56:49.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:48 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3621623527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-17T23:56:49.414 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383710 got 343597383716 for osd.11 2024-03-17T23:56:49.414 DEBUG:teuthology.parallel:result is None 2024-03-17T23:56:49.414 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-17T23:56:49.414 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-17T23:56:49.880 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:50.028 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:49 smithi012 ceph-mon[31541]: pgmap v379: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:50.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:49 smithi079 ceph-mon[36241]: pgmap v379: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:50.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:49 smithi191 ceph-mon[36708]: pgmap v379: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:51.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:51 smithi012 ceph-mon[31541]: pgmap v380: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:52.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:51 smithi079 ceph-mon[36241]: pgmap v380: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:52.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:51 smithi191 ceph-mon[36708]: pgmap v380: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:52.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.204+0000 7f0e4ea00700 1 -- 172.21.15.12:0/2748272194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e480745f0 msgr2=0x7f0e480749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:52.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.204+0000 7f0e4ea00700 1 --2- 172.21.15.12:0/2748272194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e480745f0 0x7f0e480749d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0e38009990 tx=0x7f0e38009ca0 comp rx=0 tx=0).stop 2024-03-17T23:56:52.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.204+0000 7f0e4ea00700 1 -- 172.21.15.12:0/2748272194 shutdown_connections 2024-03-17T23:56:52.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.204+0000 7f0e4ea00700 1 --2- 172.21.15.12:0/2748272194 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e4810ebb0 0x7f0e48110fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:52.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.204+0000 7f0e4ea00700 1 --2- 172.21.15.12:0/2748272194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e480745f0 0x7f0e480749d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:52.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.204+0000 7f0e4ea00700 1 --2- 172.21.15.12:0/2748272194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e48074f10 0x7f0e4810e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:52.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.204+0000 7f0e4ea00700 1 -- 172.21.15.12:0/2748272194 >> 172.21.15.12:0/2748272194 conn(0x7f0e480fdda0 msgr2=0x7f0e481001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:52.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.204+0000 7f0e4ea00700 1 -- 172.21.15.12:0/2748272194 shutdown_connections 2024-03-17T23:56:52.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.204+0000 7f0e4ea00700 1 -- 172.21.15.12:0/2748272194 wait complete. 2024-03-17T23:56:52.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.205+0000 7f0e4ea00700 1 Processor -- start 2024-03-17T23:56:52.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.205+0000 7f0e4ea00700 1 -- start start 2024-03-17T23:56:52.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e4ea00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e480745f0 0x7f0e4819ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:52.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e4ea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e48074f10 0x7f0e4819b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:52.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e4ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e4810ebb0 0x7f0e4819f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:52.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e4ea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0e48114860 con 0x7f0e4810ebb0 2024-03-17T23:56:52.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e4ea00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0e481146e0 con 0x7f0e480745f0 2024-03-17T23:56:52.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e4ea00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0e481149e0 con 0x7f0e48074f10 2024-03-17T23:56:52.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e47e00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e480745f0 0x7f0e4819ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:52.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e3fe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e48074f10 0x7f0e4819b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:52.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e47e00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e480745f0 0x7f0e4819ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:40970/0 (socket says 172.21.15.12:40970) 2024-03-17T23:56:52.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e47e00700 1 -- 172.21.15.12:0/152401851 learned_addr learned my addr 172.21.15.12:0/152401851 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:52.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e4ca00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e4810ebb0 0x7f0e4819f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:52.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e3fe00700 1 -- 172.21.15.12:0/152401851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e480745f0 msgr2=0x7f0e4819ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:52.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e3fe00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e480745f0 0x7f0e4819ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:52.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e3fe00700 1 -- 172.21.15.12:0/152401851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e4810ebb0 msgr2=0x7f0e4819f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:52.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e3fe00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e4810ebb0 0x7f0e4819f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:52.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e3fe00700 1 -- 172.21.15.12:0/152401851 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e4819fd20 con 0x7f0e48074f10 2024-03-17T23:56:52.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e47e00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e480745f0 0x7f0e4819ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:52.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.206+0000 7f0e4ca00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e4810ebb0 0x7f0e4819f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:56:52.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.207+0000 7f0e3fe00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e48074f10 0x7f0e4819b250 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000e9f0 tx=0x7f0e3000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:52.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.207+0000 7f0e45600700 1 -- 172.21.15.12:0/152401851 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3000cbd0 con 0x7f0e48074f10 2024-03-17T23:56:52.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.207+0000 7f0e45600700 1 -- 172.21.15.12:0/152401851 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e3000cd70 con 0x7f0e48074f10 2024-03-17T23:56:52.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.207+0000 7f0e45600700 1 -- 172.21.15.12:0/152401851 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e30018710 con 0x7f0e48074f10 2024-03-17T23:56:52.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.207+0000 7f0e4ea00700 1 -- 172.21.15.12:0/152401851 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e481a0010 con 0x7f0e48074f10 2024-03-17T23:56:52.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.207+0000 7f0e4ea00700 1 -- 172.21.15.12:0/152401851 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e481a7890 con 0x7f0e48074f10 2024-03-17T23:56:52.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.208+0000 7f0e4ea00700 1 -- 172.21.15.12:0/152401851 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e48068f90 con 0x7f0e48074f10 2024-03-17T23:56:52.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.212+0000 7f0e45600700 1 -- 172.21.15.12:0/152401851 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f0e30010430 con 0x7f0e48074f10 2024-03-17T23:56:52.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.213+0000 7f0e45600700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0e2c0a29a0 0x7f0e2c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:52.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.213+0000 7f0e45600700 1 -- 172.21.15.12:0/152401851 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f0e300cf1a0 con 0x7f0e48074f10 2024-03-17T23:56:52.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.214+0000 7f0e47e00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0e2c0a29a0 0x7f0e2c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.214+0000 7f0e47e00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0e2c0a29a0 0x7f0e2c0a4e60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0e38009750 tx=0x7f0e3800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:52.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.217+0000 7f0e45600700 1 -- 172.21.15.12:0/152401851 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0e3009bd20 con 0x7f0e48074f10 2024-03-17T23:56:52.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.441+0000 7f0e4ea00700 1 -- 172.21.15.12:0/152401851 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0e48062710 con 0x7f0e2c0a29a0 2024-03-17T23:56:52.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.462+0000 7f0e45600700 1 -- 172.21.15.12:0/152401851 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340720 (secure 0 0 0) 0x7f0e48062710 con 0x7f0e2c0a29a0 2024-03-17T23:56:52.470 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:56:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.471+0000 7f0e4ea00700 1 -- 172.21.15.12:0/152401851 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0e2c0a29a0 msgr2=0x7f0e2c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.471+0000 7f0e4ea00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0e2c0a29a0 0x7f0e2c0a4e60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0e38009750 tx=0x7f0e3800b280 comp rx=0 tx=0).stop 2024-03-17T23:56:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.471+0000 7f0e4ea00700 1 -- 172.21.15.12:0/152401851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e48074f10 msgr2=0x7f0e4819b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.471+0000 7f0e4ea00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e48074f10 0x7f0e4819b250 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000e9f0 tx=0x7f0e3000ed00 comp rx=0 tx=0).stop 2024-03-17T23:56:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.471+0000 7f0e4ea00700 1 -- 172.21.15.12:0/152401851 shutdown_connections 2024-03-17T23:56:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.471+0000 7f0e4ea00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0e2c0a29a0 0x7f0e2c0a4e60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.471+0000 7f0e4ea00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e4810ebb0 0x7f0e4819f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.471+0000 7f0e4ea00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e480745f0 0x7f0e4819ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.471+0000 7f0e4ea00700 1 --2- 172.21.15.12:0/152401851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e48074f10 0x7f0e4819b250 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.471+0000 7f0e4ea00700 1 -- 172.21.15.12:0/152401851 >> 172.21.15.12:0/152401851 conn(0x7f0e480fdda0 msgr2=0x7f0e480fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:52.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.471+0000 7f0e4ea00700 1 -- 172.21.15.12:0/152401851 shutdown_connections 2024-03-17T23:56:52.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:52.471+0000 7f0e4ea00700 1 -- 172.21.15.12:0/152401851 wait complete. 2024-03-17T23:56:52.474 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-17T23:56:53.349 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-03-17T23:56:52.146897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661621,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":980,"num_read_kb":615,"num_write":426,"num_write_kb":3081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1181028,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":350,"ondisk_log_size":350,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353344,"kb_used_data":12440,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588480,"statfs":{"total":1151940427776,"available":1151578603520,"internally_reserved":0,"allocated":12738560,"data_stored":8068817,"data_compressed":360444,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":120,"num_read_kb":65,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004687"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.467926+0000","last_change":"2024-03-17T23:54:56.354062+0000","last_active":"2024-03-17T23:55:55.467926+0000","last_peered":"2024-03-17T23:55:55.467926+0000","last_clean":"2024-03-17T23:55:55.467926+0000","last_became_active":"2024-03-17T23:54:56.353753+0000","last_became_peered":"2024-03-17T23:54:56.353753+0000","last_unstale":"2024-03-17T23:55:55.467926+0000","last_undegraded":"2024-03-17T23:55:55.467926+0000","last_fullsized":"2024-03-17T23:55:55.467926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:41:50.385420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.493237+0000","last_change":"2024-03-17T23:54:56.356507+0000","last_active":"2024-03-17T23:55:55.493237+0000","last_peered":"2024-03-17T23:55:55.493237+0000","last_clean":"2024-03-17T23:55:55.493237+0000","last_became_active":"2024-03-17T23:54:56.355714+0000","last_became_peered":"2024-03-17T23:54:56.355714+0000","last_unstale":"2024-03-17T23:55:55.493237+0000","last_undegraded":"2024-03-17T23:55:55.493237+0000","last_fullsized":"2024-03-17T23:55:55.493237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:44:15.992630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.494068+0000","last_change":"2024-03-17T23:54:56.352207+0000","last_active":"2024-03-17T23:55:55.494068+0000","last_peered":"2024-03-17T23:55:55.494068+0000","last_clean":"2024-03-17T23:55:55.494068+0000","last_became_active":"2024-03-17T23:54:56.351704+0000","last_became_peered":"2024-03-17T23:54:56.351704+0000","last_unstale":"2024-03-17T23:55:55.494068+0000","last_undegraded":"2024-03-17T23:55:55.494068+0000","last_fullsized":"2024-03-17T23:55:55.494068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:34.328120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470548+0000","last_change":"2024-03-17T23:54:56.343336+0000","last_active":"2024-03-17T23:55:55.470548+0000","last_peered":"2024-03-17T23:55:55.470548+0000","last_clean":"2024-03-17T23:55:55.470548+0000","last_became_active":"2024-03-17T23:54:56.343033+0000","last_became_peered":"2024-03-17T23:54:56.343033+0000","last_unstale":"2024-03-17T23:55:55.470548+0000","last_undegraded":"2024-03-17T23:55:55.470548+0000","last_fullsized":"2024-03-17T23:55:55.470548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:40:21.814558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.464679+0000","last_change":"2024-03-17T23:54:56.360541+0000","last_active":"2024-03-17T23:55:55.464679+0000","last_peered":"2024-03-17T23:55:55.464679+0000","last_clean":"2024-03-17T23:55:55.464679+0000","last_became_active":"2024-03-17T23:54:56.360204+0000","last_became_peered":"2024-03-17T23:54:56.360204+0000","last_unstale":"2024-03-17T23:55:55.464679+0000","last_undegraded":"2024-03-17T23:55:55.464679+0000","last_fullsized":"2024-03-17T23:55:55.464679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:57:37.853168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.468297+0000","last_change":"2024-03-17T23:54:56.363727+0000","last_active":"2024-03-17T23:55:55.468297+0000","last_peered":"2024-03-17T23:55:55.468297+0000","last_clean":"2024-03-17T23:55:55.468297+0000","last_became_active":"2024-03-17T23:54:56.363399+0000","last_became_peered":"2024-03-17T23:54:56.363399+0000","last_unstale":"2024-03-17T23:55:55.468297+0000","last_undegraded":"2024-03-17T23:55:55.468297+0000","last_fullsized":"2024-03-17T23:55:55.468297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:42.451379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.570468+0000","last_change":"2024-03-17T23:54:56.361881+0000","last_active":"2024-03-17T23:55:55.570468+0000","last_peered":"2024-03-17T23:55:55.570468+0000","last_clean":"2024-03-17T23:55:55.570468+0000","last_became_active":"2024-03-17T23:54:56.361531+0000","last_became_peered":"2024-03-17T23:54:56.361531+0000","last_unstale":"2024-03-17T23:55:55.570468+0000","last_undegraded":"2024-03-17T23:55:55.570468+0000","last_fullsized":"2024-03-17T23:55:55.570468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:44:39.792763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.480556+0000","last_change":"2024-03-17T23:54:56.358429+0000","last_active":"2024-03-17T23:55:55.480556+0000","last_peered":"2024-03-17T23:55:55.480556+0000","last_clean":"2024-03-17T23:55:55.480556+0000","last_became_active":"2024-03-17T23:54:56.355268+0000","last_became_peered":"2024-03-17T23:54:56.355268+0000","last_unstale":"2024-03-17T23:55:55.480556+0000","last_undegraded":"2024-03-17T23:55:55.480556+0000","last_fullsized":"2024-03-17T23:55:55.480556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:00:37.687542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571994+0000","last_change":"2024-03-17T23:54:56.344228+0000","last_active":"2024-03-17T23:55:55.571994+0000","last_peered":"2024-03-17T23:55:55.571994+0000","last_clean":"2024-03-17T23:55:55.571994+0000","last_became_active":"2024-03-17T23:54:56.343863+0000","last_became_peered":"2024-03-17T23:54:56.343863+0000","last_unstale":"2024-03-17T23:55:55.571994+0000","last_undegraded":"2024-03-17T23:55:55.571994+0000","last_fullsized":"2024-03-17T23:55:55.571994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:45:12.552611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.468587+0000","last_change":"2024-03-17T23:54:56.341215+0000","last_active":"2024-03-17T23:55:55.468587+0000","last_peered":"2024-03-17T23:55:55.468587+0000","last_clean":"2024-03-17T23:55:55.468587+0000","last_became_active":"2024-03-17T23:54:56.340850+0000","last_became_peered":"2024-03-17T23:54:56.340850+0000","last_unstale":"2024-03-17T23:55:55.468587+0000","last_undegraded":"2024-03-17T23:55:55.468587+0000","last_fullsized":"2024-03-17T23:55:55.468587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:28.273320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571564+0000","last_change":"2024-03-17T23:54:56.346613+0000","last_active":"2024-03-17T23:55:55.571564+0000","last_peered":"2024-03-17T23:55:55.571564+0000","last_clean":"2024-03-17T23:55:55.571564+0000","last_became_active":"2024-03-17T23:54:56.345907+0000","last_became_peered":"2024-03-17T23:54:56.345907+0000","last_unstale":"2024-03-17T23:55:55.571564+0000","last_undegraded":"2024-03-17T23:55:55.571564+0000","last_fullsized":"2024-03-17T23:55:55.571564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:21.493841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470351+0000","last_change":"2024-03-17T23:54:56.341924+0000","last_active":"2024-03-17T23:55:55.470351+0000","last_peered":"2024-03-17T23:55:55.470351+0000","last_clean":"2024-03-17T23:55:55.470351+0000","last_became_active":"2024-03-17T23:54:56.341469+0000","last_became_peered":"2024-03-17T23:54:56.341469+0000","last_unstale":"2024-03-17T23:55:55.470351+0000","last_undegraded":"2024-03-17T23:55:55.470351+0000","last_fullsized":"2024-03-17T23:55:55.470351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:49:47.256954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.477694+0000","last_change":"2024-03-17T23:54:56.332469+0000","last_active":"2024-03-17T23:55:55.477694+0000","last_peered":"2024-03-17T23:55:55.477694+0000","last_clean":"2024-03-17T23:55:55.477694+0000","last_became_active":"2024-03-17T23:54:56.330684+0000","last_became_peered":"2024-03-17T23:54:56.330684+0000","last_unstale":"2024-03-17T23:55:55.477694+0000","last_undegraded":"2024-03-17T23:55:55.477694+0000","last_fullsized":"2024-03-17T23:55:55.477694+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:25.211711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.480680+0000","last_change":"2024-03-17T23:54:56.354451+0000","last_active":"2024-03-17T23:55:55.480680+0000","last_peered":"2024-03-17T23:55:55.480680+0000","last_clean":"2024-03-17T23:55:55.480680+0000","last_became_active":"2024-03-17T23:54:56.354160+0000","last_became_peered":"2024-03-17T23:54:56.354160+0000","last_unstale":"2024-03-17T23:55:55.480680+0000","last_undegraded":"2024-03-17T23:55:55.480680+0000","last_fullsized":"2024-03-17T23:55:55.480680+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:07:37.644444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.486979+0000","last_change":"2024-03-17T23:54:56.353705+0000","last_active":"2024-03-17T23:55:55.486979+0000","last_peered":"2024-03-17T23:55:55.486979+0000","last_clean":"2024-03-17T23:55:55.486979+0000","last_became_active":"2024-03-17T23:54:56.352389+0000","last_became_peered":"2024-03-17T23:54:56.352389+0000","last_unstale":"2024-03-17T23:55:55.486979+0000","last_undegraded":"2024-03-17T23:55:55.486979+0000","last_fullsized":"2024-03-17T23:55:55.486979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:17.153753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.570930+0000","last_change":"2024-03-17T23:54:56.348467+0000","last_active":"2024-03-17T23:55:55.570930+0000","last_peered":"2024-03-17T23:55:55.570930+0000","last_clean":"2024-03-17T23:55:55.570930+0000","last_became_active":"2024-03-17T23:54:56.348118+0000","last_became_peered":"2024-03-17T23:54:56.348118+0000","last_unstale":"2024-03-17T23:55:55.570930+0000","last_undegraded":"2024-03-17T23:55:55.570930+0000","last_fullsized":"2024-03-17T23:55:55.570930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:53:45.717303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.467820+0000","last_change":"2024-03-17T23:54:56.354117+0000","last_active":"2024-03-17T23:55:55.467820+0000","last_peered":"2024-03-17T23:55:55.467820+0000","last_clean":"2024-03-17T23:55:55.467820+0000","last_became_active":"2024-03-17T23:54:56.353818+0000","last_became_peered":"2024-03-17T23:54:56.353818+0000","last_unstale":"2024-03-17T23:55:55.467820+0000","last_undegraded":"2024-03-17T23:55:55.467820+0000","last_fullsized":"2024-03-17T23:55:55.467820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:09:17.314024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.501154+0000","last_change":"2024-03-17T23:54:56.363781+0000","last_active":"2024-03-17T23:55:55.501154+0000","last_peered":"2024-03-17T23:55:55.501154+0000","last_clean":"2024-03-17T23:55:55.501154+0000","last_became_active":"2024-03-17T23:54:56.363364+0000","last_became_peered":"2024-03-17T23:54:56.363364+0000","last_unstale":"2024-03-17T23:55:55.501154+0000","last_undegraded":"2024-03-17T23:55:55.501154+0000","last_fullsized":"2024-03-17T23:55:55.501154+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:21:00.395329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.467216+0000","last_change":"2024-03-17T23:54:56.360905+0000","last_active":"2024-03-17T23:55:55.467216+0000","last_peered":"2024-03-17T23:55:55.467216+0000","last_clean":"2024-03-17T23:55:55.467216+0000","last_became_active":"2024-03-17T23:54:56.360492+0000","last_became_peered":"2024-03-17T23:54:56.360492+0000","last_unstale":"2024-03-17T23:55:55.467216+0000","last_undegraded":"2024-03-17T23:55:55.467216+0000","last_fullsized":"2024-03-17T23:55:55.467216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:34:42.440635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466685+0000","last_change":"2024-03-17T23:55:01.889086+0000","last_active":"2024-03-17T23:55:55.466685+0000","last_peered":"2024-03-17T23:55:55.466685+0000","last_clean":"2024-03-17T23:55:55.466685+0000","last_became_active":"2024-03-17T23:55:01.888505+0000","last_became_peered":"2024-03-17T23:55:01.888505+0000","last_unstale":"2024-03-17T23:55:55.466685+0000","last_undegraded":"2024-03-17T23:55:55.466685+0000","last_fullsized":"2024-03-17T23:55:55.466685+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:00:43.490360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":125,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:48.166066+0000","last_change":"2024-03-17T23:54:55.858790+0000","last_active":"2024-03-17T23:56:48.166066+0000","last_peered":"2024-03-17T23:56:48.166066+0000","last_clean":"2024-03-17T23:56:48.166066+0000","last_became_active":"2024-03-17T23:54:55.858437+0000","last_became_peered":"2024-03-17T23:54:55.858437+0000","last_unstale":"2024-03-17T23:56:48.166066+0000","last_undegraded":"2024-03-17T23:56:48.166066+0000","last_fullsized":"2024-03-17T23:56:48.166066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:49:26.830471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.482904+0000","last_change":"2024-03-17T23:54:56.356641+0000","last_active":"2024-03-17T23:55:55.482904+0000","last_peered":"2024-03-17T23:55:55.482904+0000","last_clean":"2024-03-17T23:55:55.482904+0000","last_became_active":"2024-03-17T23:54:56.355402+0000","last_became_peered":"2024-03-17T23:54:56.355402+0000","last_unstale":"2024-03-17T23:55:55.482904+0000","last_undegraded":"2024-03-17T23:55:55.482904+0000","last_fullsized":"2024-03-17T23:55:55.482904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:01:04.699357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470595+0000","last_change":"2024-03-17T23:55:01.888632+0000","last_active":"2024-03-17T23:55:55.470595+0000","last_peered":"2024-03-17T23:55:55.470595+0000","last_clean":"2024-03-17T23:55:55.470595+0000","last_became_active":"2024-03-17T23:55:01.888240+0000","last_became_peered":"2024-03-17T23:55:01.888240+0000","last_unstale":"2024-03-17T23:55:55.470595+0000","last_undegraded":"2024-03-17T23:55:55.470595+0000","last_fullsized":"2024-03-17T23:55:55.470595+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:02:04.562112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.483028+0000","last_change":"2024-03-17T23:54:55.871842+0000","last_active":"2024-03-17T23:55:55.483028+0000","last_peered":"2024-03-17T23:55:55.483028+0000","last_clean":"2024-03-17T23:55:55.483028+0000","last_became_active":"2024-03-17T23:54:55.870965+0000","last_became_peered":"2024-03-17T23:54:55.870965+0000","last_unstale":"2024-03-17T23:55:55.483028+0000","last_undegraded":"2024-03-17T23:55:55.483028+0000","last_fullsized":"2024-03-17T23:55:55.483028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:27:04.762902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.495217+0000","last_change":"2024-03-17T23:54:56.368277+0000","last_active":"2024-03-17T23:55:55.495217+0000","last_peered":"2024-03-17T23:55:55.495217+0000","last_clean":"2024-03-17T23:55:55.495217+0000","last_became_active":"2024-03-17T23:54:56.366229+0000","last_became_peered":"2024-03-17T23:54:56.366229+0000","last_unstale":"2024-03-17T23:55:55.495217+0000","last_undegraded":"2024-03-17T23:55:55.495217+0000","last_fullsized":"2024-03-17T23:55:55.495217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:50:05.241745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.469885+0000","last_change":"2024-03-17T23:55:01.881731+0000","last_active":"2024-03-17T23:55:55.469885+0000","last_peered":"2024-03-17T23:55:55.469885+0000","last_clean":"2024-03-17T23:55:55.469885+0000","last_became_active":"2024-03-17T23:55:01.881383+0000","last_became_peered":"2024-03-17T23:55:01.881383+0000","last_unstale":"2024-03-17T23:55:55.469885+0000","last_undegraded":"2024-03-17T23:55:55.469885+0000","last_fullsized":"2024-03-17T23:55:55.469885+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:45.095672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.465103+0000","last_change":"2024-03-17T23:54:55.838014+0000","last_active":"2024-03-17T23:55:55.465103+0000","last_peered":"2024-03-17T23:55:55.465103+0000","last_clean":"2024-03-17T23:55:55.465103+0000","last_became_active":"2024-03-17T23:54:55.837694+0000","last_became_peered":"2024-03-17T23:54:55.837694+0000","last_unstale":"2024-03-17T23:55:55.465103+0000","last_undegraded":"2024-03-17T23:55:55.465103+0000","last_fullsized":"2024-03-17T23:55:55.465103+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:16:01.173460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571645+0000","last_change":"2024-03-17T23:54:56.343240+0000","last_active":"2024-03-17T23:55:55.571645+0000","last_peered":"2024-03-17T23:55:55.571645+0000","last_clean":"2024-03-17T23:55:55.571645+0000","last_became_active":"2024-03-17T23:54:56.342928+0000","last_became_peered":"2024-03-17T23:54:56.342928+0000","last_unstale":"2024-03-17T23:55:55.571645+0000","last_undegraded":"2024-03-17T23:55:55.571645+0000","last_fullsized":"2024-03-17T23:55:55.571645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:33:43.922886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.469658+0000","last_change":"2024-03-17T23:55:01.880207+0000","last_active":"2024-03-17T23:55:55.469658+0000","last_peered":"2024-03-17T23:55:55.469658+0000","last_clean":"2024-03-17T23:55:55.469658+0000","last_became_active":"2024-03-17T23:55:01.879748+0000","last_became_peered":"2024-03-17T23:55:01.879748+0000","last_unstale":"2024-03-17T23:55:55.469658+0000","last_undegraded":"2024-03-17T23:55:55.469658+0000","last_fullsized":"2024-03-17T23:55:55.469658+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:33.782782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.464916+0000","last_change":"2024-03-17T23:54:55.839822+0000","last_active":"2024-03-17T23:55:55.464916+0000","last_peered":"2024-03-17T23:55:55.464916+0000","last_clean":"2024-03-17T23:55:55.464916+0000","last_became_active":"2024-03-17T23:54:55.839506+0000","last_became_peered":"2024-03-17T23:54:55.839506+0000","last_unstale":"2024-03-17T23:55:55.464916+0000","last_undegraded":"2024-03-17T23:55:55.464916+0000","last_fullsized":"2024-03-17T23:55:55.464916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:08.743582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470977+0000","last_change":"2024-03-17T23:54:56.346914+0000","last_active":"2024-03-17T23:55:55.470977+0000","last_peered":"2024-03-17T23:55:55.470977+0000","last_clean":"2024-03-17T23:55:55.470977+0000","last_became_active":"2024-03-17T23:54:56.346620+0000","last_became_peered":"2024-03-17T23:54:56.346620+0000","last_unstale":"2024-03-17T23:55:55.470977+0000","last_undegraded":"2024-03-17T23:55:55.470977+0000","last_fullsized":"2024-03-17T23:55:55.470977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:44:58.643614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.467093+0000","last_change":"2024-03-17T23:55:01.864527+0000","last_active":"2024-03-17T23:55:55.467093+0000","last_peered":"2024-03-17T23:55:55.467093+0000","last_clean":"2024-03-17T23:55:55.467093+0000","last_became_active":"2024-03-17T23:55:01.864106+0000","last_became_peered":"2024-03-17T23:55:01.864106+0000","last_unstale":"2024-03-17T23:55:55.467093+0000","last_undegraded":"2024-03-17T23:55:55.467093+0000","last_fullsized":"2024-03-17T23:55:55.467093+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:27:19.811639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":116,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:46.122794+0000","last_change":"2024-03-17T23:54:55.842156+0000","last_active":"2024-03-17T23:56:46.122794+0000","last_peered":"2024-03-17T23:56:46.122794+0000","last_clean":"2024-03-17T23:56:46.122794+0000","last_became_active":"2024-03-17T23:54:55.841858+0000","last_became_peered":"2024-03-17T23:54:55.841858+0000","last_unstale":"2024-03-17T23:56:46.122794+0000","last_undegraded":"2024-03-17T23:56:46.122794+0000","last_fullsized":"2024-03-17T23:56:46.122794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:01:05.967638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.465947+0000","last_change":"2024-03-17T23:54:56.357770+0000","last_active":"2024-03-17T23:55:55.465947+0000","last_peered":"2024-03-17T23:55:55.465947+0000","last_clean":"2024-03-17T23:55:55.465947+0000","last_became_active":"2024-03-17T23:54:56.357411+0000","last_became_peered":"2024-03-17T23:54:56.357411+0000","last_unstale":"2024-03-17T23:55:55.465947+0000","last_undegraded":"2024-03-17T23:55:55.465947+0000","last_fullsized":"2024-03-17T23:55:55.465947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:01:04.685138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.467219+0000","last_change":"2024-03-17T23:55:01.883104+0000","last_active":"2024-03-17T23:55:55.467219+0000","last_peered":"2024-03-17T23:55:55.467219+0000","last_clean":"2024-03-17T23:55:55.467219+0000","last_became_active":"2024-03-17T23:55:01.882650+0000","last_became_peered":"2024-03-17T23:55:01.882650+0000","last_unstale":"2024-03-17T23:55:55.467219+0000","last_undegraded":"2024-03-17T23:55:55.467219+0000","last_fullsized":"2024-03-17T23:55:55.467219+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:54:18.420997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.576887+0000","last_change":"2024-03-17T23:55:55.576887+0000","last_active":"2024-03-17T23:55:55.576887+0000","last_peered":"2024-03-17T23:55:55.576887+0000","last_clean":"2024-03-17T23:55:55.576887+0000","last_became_active":"2024-03-17T23:55:55.572572+0000","last_became_peered":"2024-03-17T23:55:55.572572+0000","last_unstale":"2024-03-17T23:55:55.576887+0000","last_undegraded":"2024-03-17T23:55:55.576887+0000","last_fullsized":"2024-03-17T23:55:55.576887+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:47:44.693051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470933+0000","last_change":"2024-03-17T23:54:56.353311+0000","last_active":"2024-03-17T23:55:55.470933+0000","last_peered":"2024-03-17T23:55:55.470933+0000","last_clean":"2024-03-17T23:55:55.470933+0000","last_became_active":"2024-03-17T23:54:56.353013+0000","last_became_peered":"2024-03-17T23:54:56.353013+0000","last_unstale":"2024-03-17T23:55:55.470933+0000","last_undegraded":"2024-03-17T23:55:55.470933+0000","last_fullsized":"2024-03-17T23:55:55.470933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:21:20.119186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.497661+0000","last_change":"2024-03-17T23:55:55.497661+0000","last_active":"2024-03-17T23:55:55.497661+0000","last_peered":"2024-03-17T23:55:55.497661+0000","last_clean":"2024-03-17T23:55:55.497661+0000","last_became_active":"2024-03-17T23:55:55.497263+0000","last_became_peered":"2024-03-17T23:55:55.497263+0000","last_unstale":"2024-03-17T23:55:55.497661+0000","last_undegraded":"2024-03-17T23:55:55.497661+0000","last_fullsized":"2024-03-17T23:55:55.497661+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:15:23.555869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.481012+0000","last_change":"2024-03-17T23:54:55.862756+0000","last_active":"2024-03-17T23:55:55.481012+0000","last_peered":"2024-03-17T23:55:55.481012+0000","last_clean":"2024-03-17T23:55:55.481012+0000","last_became_active":"2024-03-17T23:54:55.862298+0000","last_became_peered":"2024-03-17T23:54:55.862298+0000","last_unstale":"2024-03-17T23:55:55.481012+0000","last_undegraded":"2024-03-17T23:55:55.481012+0000","last_fullsized":"2024-03-17T23:55:55.481012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:15:52.613136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.480429+0000","last_change":"2024-03-17T23:54:56.354359+0000","last_active":"2024-03-17T23:55:55.480429+0000","last_peered":"2024-03-17T23:55:55.480429+0000","last_clean":"2024-03-17T23:55:55.480429+0000","last_became_active":"2024-03-17T23:54:56.354002+0000","last_became_peered":"2024-03-17T23:54:56.354002+0000","last_unstale":"2024-03-17T23:55:55.480429+0000","last_undegraded":"2024-03-17T23:55:55.480429+0000","last_fullsized":"2024-03-17T23:55:55.480429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:41:12.128471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.493005+0000","last_change":"2024-03-17T23:55:01.880579+0000","last_active":"2024-03-17T23:55:55.493005+0000","last_peered":"2024-03-17T23:55:55.493005+0000","last_clean":"2024-03-17T23:55:55.493005+0000","last_became_active":"2024-03-17T23:55:01.879471+0000","last_became_peered":"2024-03-17T23:55:01.879471+0000","last_unstale":"2024-03-17T23:55:55.493005+0000","last_undegraded":"2024-03-17T23:55:55.493005+0000","last_fullsized":"2024-03-17T23:55:55.493005+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:51.117904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470548+0000","last_change":"2024-03-17T23:54:55.842998+0000","last_active":"2024-03-17T23:55:55.470548+0000","last_peered":"2024-03-17T23:55:55.470548+0000","last_clean":"2024-03-17T23:55:55.470548+0000","last_became_active":"2024-03-17T23:54:55.842678+0000","last_became_peered":"2024-03-17T23:54:55.842678+0000","last_unstale":"2024-03-17T23:55:55.470548+0000","last_undegraded":"2024-03-17T23:55:55.470548+0000","last_fullsized":"2024-03-17T23:55:55.470548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:14.546738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.495807+0000","last_change":"2024-03-17T23:54:56.365881+0000","last_active":"2024-03-17T23:55:55.495807+0000","last_peered":"2024-03-17T23:55:55.495807+0000","last_clean":"2024-03-17T23:55:55.495807+0000","last_became_active":"2024-03-17T23:54:56.365560+0000","last_became_peered":"2024-03-17T23:54:56.365560+0000","last_unstale":"2024-03-17T23:55:55.495807+0000","last_undegraded":"2024-03-17T23:55:55.495807+0000","last_fullsized":"2024-03-17T23:55:55.495807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:38.436644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.481311+0000","last_change":"2024-03-17T23:55:01.882651+0000","last_active":"2024-03-17T23:55:55.481311+0000","last_peered":"2024-03-17T23:55:55.481311+0000","last_clean":"2024-03-17T23:55:55.481311+0000","last_became_active":"2024-03-17T23:55:01.881717+0000","last_became_peered":"2024-03-17T23:55:01.881717+0000","last_unstale":"2024-03-17T23:55:55.481311+0000","last_undegraded":"2024-03-17T23:55:55.481311+0000","last_fullsized":"2024-03-17T23:55:55.481311+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:21:08.644184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":199,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:50.126380+0000","last_change":"2024-03-17T23:54:55.854822+0000","last_active":"2024-03-17T23:56:50.126380+0000","last_peered":"2024-03-17T23:56:50.126380+0000","last_clean":"2024-03-17T23:56:50.126380+0000","last_became_active":"2024-03-17T23:54:55.854501+0000","last_became_peered":"2024-03-17T23:54:55.854501+0000","last_unstale":"2024-03-17T23:56:50.126380+0000","last_undegraded":"2024-03-17T23:56:50.126380+0000","last_fullsized":"2024-03-17T23:56:50.126380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:04:12.504798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":166,"num_read_kb":166,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571957+0000","last_change":"2024-03-17T23:54:56.361898+0000","last_active":"2024-03-17T23:55:55.571957+0000","last_peered":"2024-03-17T23:55:55.571957+0000","last_clean":"2024-03-17T23:55:55.571957+0000","last_became_active":"2024-03-17T23:54:56.361556+0000","last_became_peered":"2024-03-17T23:54:56.361556+0000","last_unstale":"2024-03-17T23:55:55.571957+0000","last_undegraded":"2024-03-17T23:55:55.571957+0000","last_fullsized":"2024-03-17T23:55:55.571957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:52:08.932801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.491562+0000","last_change":"2024-03-17T23:55:01.882628+0000","last_active":"2024-03-17T23:55:55.491562+0000","last_peered":"2024-03-17T23:55:55.491562+0000","last_clean":"2024-03-17T23:55:55.491562+0000","last_became_active":"2024-03-17T23:55:01.881886+0000","last_became_peered":"2024-03-17T23:55:01.881886+0000","last_unstale":"2024-03-17T23:55:55.491562+0000","last_undegraded":"2024-03-17T23:55:55.491562+0000","last_fullsized":"2024-03-17T23:55:55.491562+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:22:58.534442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571317+0000","last_change":"2024-03-17T23:54:55.839925+0000","last_active":"2024-03-17T23:55:55.571317+0000","last_peered":"2024-03-17T23:55:55.571317+0000","last_clean":"2024-03-17T23:55:55.571317+0000","last_became_active":"2024-03-17T23:54:55.839626+0000","last_became_peered":"2024-03-17T23:54:55.839626+0000","last_unstale":"2024-03-17T23:55:55.571317+0000","last_undegraded":"2024-03-17T23:55:55.571317+0000","last_fullsized":"2024-03-17T23:55:55.571317+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:39.456462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.494733+0000","last_change":"2024-03-17T23:54:56.352766+0000","last_active":"2024-03-17T23:55:55.494733+0000","last_peered":"2024-03-17T23:55:55.494733+0000","last_clean":"2024-03-17T23:55:55.494733+0000","last_became_active":"2024-03-17T23:54:56.351817+0000","last_became_peered":"2024-03-17T23:54:56.351817+0000","last_unstale":"2024-03-17T23:55:55.494733+0000","last_undegraded":"2024-03-17T23:55:55.494733+0000","last_fullsized":"2024-03-17T23:55:55.494733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:57:38.097163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.572538+0000","last_change":"2024-03-17T23:55:55.572538+0000","last_active":"2024-03-17T23:55:55.572538+0000","last_peered":"2024-03-17T23:55:55.572538+0000","last_clean":"2024-03-17T23:55:55.572538+0000","last_became_active":"2024-03-17T23:55:55.572138+0000","last_became_peered":"2024-03-17T23:55:55.572138+0000","last_unstale":"2024-03-17T23:55:55.572538+0000","last_undegraded":"2024-03-17T23:55:55.572538+0000","last_fullsized":"2024-03-17T23:55:55.572538+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:20:39.628674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470989+0000","last_change":"2024-03-17T23:54:55.863264+0000","last_active":"2024-03-17T23:55:55.470989+0000","last_peered":"2024-03-17T23:55:55.470989+0000","last_clean":"2024-03-17T23:55:55.470989+0000","last_became_active":"2024-03-17T23:54:55.863000+0000","last_became_peered":"2024-03-17T23:54:55.863000+0000","last_unstale":"2024-03-17T23:55:55.470989+0000","last_undegraded":"2024-03-17T23:55:55.470989+0000","last_fullsized":"2024-03-17T23:55:55.470989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:58.208064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571822+0000","last_change":"2024-03-17T23:54:56.351161+0000","last_active":"2024-03-17T23:55:55.571822+0000","last_peered":"2024-03-17T23:55:55.571822+0000","last_clean":"2024-03-17T23:55:55.571822+0000","last_became_active":"2024-03-17T23:54:56.349787+0000","last_became_peered":"2024-03-17T23:54:56.349787+0000","last_unstale":"2024-03-17T23:55:55.571822+0000","last_undegraded":"2024-03-17T23:55:55.571822+0000","last_fullsized":"2024-03-17T23:55:55.571822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:19:28.635060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.500462+0000","last_change":"2024-03-17T23:55:55.500462+0000","last_active":"2024-03-17T23:55:55.500462+0000","last_peered":"2024-03-17T23:55:55.500462+0000","last_clean":"2024-03-17T23:55:55.500462+0000","last_became_active":"2024-03-17T23:55:55.499357+0000","last_became_peered":"2024-03-17T23:55:55.499357+0000","last_unstale":"2024-03-17T23:55:55.500462+0000","last_undegraded":"2024-03-17T23:55:55.500462+0000","last_fullsized":"2024-03-17T23:55:55.500462+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:18:28.701636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":237,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:50.126383+0000","last_change":"2024-03-17T23:54:55.846449+0000","last_active":"2024-03-17T23:56:50.126383+0000","last_peered":"2024-03-17T23:56:50.126383+0000","last_clean":"2024-03-17T23:56:50.126383+0000","last_became_active":"2024-03-17T23:54:55.846064+0000","last_became_peered":"2024-03-17T23:54:55.846064+0000","last_unstale":"2024-03-17T23:56:50.126383+0000","last_undegraded":"2024-03-17T23:56:50.126383+0000","last_fullsized":"2024-03-17T23:56:50.126383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:45.985724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64206,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":173,"num_read_kb":149,"num_write":37,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.464816+0000","last_change":"2024-03-17T23:54:56.343717+0000","last_active":"2024-03-17T23:55:55.464816+0000","last_peered":"2024-03-17T23:55:55.464816+0000","last_clean":"2024-03-17T23:55:55.464816+0000","last_became_active":"2024-03-17T23:54:56.343418+0000","last_became_peered":"2024-03-17T23:54:56.343418+0000","last_unstale":"2024-03-17T23:55:55.464816+0000","last_undegraded":"2024-03-17T23:55:55.464816+0000","last_fullsized":"2024-03-17T23:55:55.464816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:11:40.402878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.471237+0000","last_change":"2024-03-17T23:55:01.884671+0000","last_active":"2024-03-17T23:55:55.471237+0000","last_peered":"2024-03-17T23:55:55.471237+0000","last_clean":"2024-03-17T23:55:55.471237+0000","last_became_active":"2024-03-17T23:55:01.884102+0000","last_became_peered":"2024-03-17T23:55:01.884102+0000","last_unstale":"2024-03-17T23:55:55.471237+0000","last_undegraded":"2024-03-17T23:55:55.471237+0000","last_fullsized":"2024-03-17T23:55:55.471237+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:53:31.198466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.504358+0000","last_change":"2024-03-17T23:55:55.503973+0000","last_active":"2024-03-17T23:55:55.504358+0000","last_peered":"2024-03-17T23:55:55.504358+0000","last_clean":"2024-03-17T23:55:55.504358+0000","last_became_active":"2024-03-17T23:55:55.488988+0000","last_became_peered":"2024-03-17T23:55:55.488988+0000","last_unstale":"2024-03-17T23:55:55.504358+0000","last_undegraded":"2024-03-17T23:55:55.504358+0000","last_fullsized":"2024-03-17T23:55:55.504358+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:42:13.924400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470807+0000","last_change":"2024-03-17T23:54:56.361699+0000","last_active":"2024-03-17T23:55:55.470807+0000","last_peered":"2024-03-17T23:55:55.470807+0000","last_clean":"2024-03-17T23:55:55.470807+0000","last_became_active":"2024-03-17T23:54:56.361365+0000","last_became_peered":"2024-03-17T23:54:56.361365+0000","last_unstale":"2024-03-17T23:55:55.470807+0000","last_undegraded":"2024-03-17T23:55:55.470807+0000","last_fullsized":"2024-03-17T23:55:55.470807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:24:30.649956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466789+0000","last_change":"2024-03-17T23:55:01.888676+0000","last_active":"2024-03-17T23:55:55.466789+0000","last_peered":"2024-03-17T23:55:55.466789+0000","last_clean":"2024-03-17T23:55:55.466789+0000","last_became_active":"2024-03-17T23:55:01.888224+0000","last_became_peered":"2024-03-17T23:55:01.888224+0000","last_unstale":"2024-03-17T23:55:55.466789+0000","last_undegraded":"2024-03-17T23:55:55.466789+0000","last_fullsized":"2024-03-17T23:55:55.466789+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:22.075096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470689+0000","last_change":"2024-03-17T23:54:55.857947+0000","last_active":"2024-03-17T23:55:55.470689+0000","last_peered":"2024-03-17T23:55:55.470689+0000","last_clean":"2024-03-17T23:55:55.470689+0000","last_became_active":"2024-03-17T23:54:55.857595+0000","last_became_peered":"2024-03-17T23:54:55.857595+0000","last_unstale":"2024-03-17T23:55:55.470689+0000","last_undegraded":"2024-03-17T23:55:55.470689+0000","last_fullsized":"2024-03-17T23:55:55.470689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:25.433053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571216+0000","last_change":"2024-03-17T23:54:56.347220+0000","last_active":"2024-03-17T23:55:55.571216+0000","last_peered":"2024-03-17T23:55:55.571216+0000","last_clean":"2024-03-17T23:55:55.571216+0000","last_became_active":"2024-03-17T23:54:56.346600+0000","last_became_peered":"2024-03-17T23:54:56.346600+0000","last_unstale":"2024-03-17T23:55:55.571216+0000","last_undegraded":"2024-03-17T23:55:55.571216+0000","last_fullsized":"2024-03-17T23:55:55.571216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:38:25.575448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.496996+0000","last_change":"2024-03-17T23:55:01.887016+0000","last_active":"2024-03-17T23:55:55.496996+0000","last_peered":"2024-03-17T23:55:55.496996+0000","last_clean":"2024-03-17T23:55:55.496996+0000","last_became_active":"2024-03-17T23:55:01.886576+0000","last_became_peered":"2024-03-17T23:55:01.886576+0000","last_unstale":"2024-03-17T23:55:55.496996+0000","last_undegraded":"2024-03-17T23:55:55.496996+0000","last_fullsized":"2024-03-17T23:55:55.496996+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:41:26.995982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":118,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:48.196662+0000","last_change":"2024-03-17T23:54:55.840254+0000","last_active":"2024-03-17T23:56:48.196662+0000","last_peered":"2024-03-17T23:56:48.196662+0000","last_clean":"2024-03-17T23:56:48.196662+0000","last_became_active":"2024-03-17T23:54:55.839740+0000","last_became_peered":"2024-03-17T23:54:55.839740+0000","last_unstale":"2024-03-17T23:56:48.196662+0000","last_undegraded":"2024-03-17T23:56:48.196662+0000","last_fullsized":"2024-03-17T23:56:48.196662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:39.664083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.570499+0000","last_change":"2024-03-17T23:54:56.343262+0000","last_active":"2024-03-17T23:55:55.570499+0000","last_peered":"2024-03-17T23:55:55.570499+0000","last_clean":"2024-03-17T23:55:55.570499+0000","last_became_active":"2024-03-17T23:54:56.342924+0000","last_became_peered":"2024-03-17T23:54:56.342924+0000","last_unstale":"2024-03-17T23:55:55.570499+0000","last_undegraded":"2024-03-17T23:55:55.570499+0000","last_fullsized":"2024-03-17T23:55:55.570499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:35:19.074001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.476494+0000","last_change":"2024-03-17T23:55:01.885081+0000","last_active":"2024-03-17T23:55:55.476494+0000","last_peered":"2024-03-17T23:55:55.476494+0000","last_clean":"2024-03-17T23:55:55.476494+0000","last_became_active":"2024-03-17T23:55:01.883328+0000","last_became_peered":"2024-03-17T23:55:01.883328+0000","last_unstale":"2024-03-17T23:55:55.476494+0000","last_undegraded":"2024-03-17T23:55:55.476494+0000","last_fullsized":"2024-03-17T23:55:55.476494+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:12:10.206774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.510896+0000","last_change":"2024-03-17T23:55:55.510896+0000","last_active":"2024-03-17T23:55:55.510896+0000","last_peered":"2024-03-17T23:55:55.510896+0000","last_clean":"2024-03-17T23:55:55.510896+0000","last_became_active":"2024-03-17T23:55:55.494373+0000","last_became_peered":"2024-03-17T23:55:55.494373+0000","last_unstale":"2024-03-17T23:55:55.510896+0000","last_undegraded":"2024-03-17T23:55:55.510896+0000","last_fullsized":"2024-03-17T23:55:55.510896+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:28:33.215554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470605+0000","last_change":"2024-03-17T23:54:56.345946+0000","last_active":"2024-03-17T23:55:55.470605+0000","last_peered":"2024-03-17T23:55:55.470605+0000","last_clean":"2024-03-17T23:55:55.470605+0000","last_became_active":"2024-03-17T23:54:56.345620+0000","last_became_peered":"2024-03-17T23:54:56.345620+0000","last_unstale":"2024-03-17T23:55:55.470605+0000","last_undegraded":"2024-03-17T23:55:55.470605+0000","last_fullsized":"2024-03-17T23:55:55.470605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:03:51.327966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470512+0000","last_change":"2024-03-17T23:55:01.884819+0000","last_active":"2024-03-17T23:55:55.470512+0000","last_peered":"2024-03-17T23:55:55.470512+0000","last_clean":"2024-03-17T23:55:55.470512+0000","last_became_active":"2024-03-17T23:55:01.884045+0000","last_became_peered":"2024-03-17T23:55:01.884045+0000","last_unstale":"2024-03-17T23:55:55.470512+0000","last_undegraded":"2024-03-17T23:55:55.470512+0000","last_fullsized":"2024-03-17T23:55:55.470512+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:10.147132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.465237+0000","last_change":"2024-03-17T23:54:55.840370+0000","last_active":"2024-03-17T23:55:55.465237+0000","last_peered":"2024-03-17T23:55:55.465237+0000","last_clean":"2024-03-17T23:55:55.465237+0000","last_became_active":"2024-03-17T23:54:55.840049+0000","last_became_peered":"2024-03-17T23:54:55.840049+0000","last_unstale":"2024-03-17T23:55:55.465237+0000","last_undegraded":"2024-03-17T23:55:55.465237+0000","last_fullsized":"2024-03-17T23:55:55.465237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:40:46.523632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466733+0000","last_change":"2024-03-17T23:54:56.346533+0000","last_active":"2024-03-17T23:55:55.466733+0000","last_peered":"2024-03-17T23:55:55.466733+0000","last_clean":"2024-03-17T23:55:55.466733+0000","last_became_active":"2024-03-17T23:54:56.346214+0000","last_became_peered":"2024-03-17T23:54:56.346214+0000","last_unstale":"2024-03-17T23:55:55.466733+0000","last_undegraded":"2024-03-17T23:55:55.466733+0000","last_fullsized":"2024-03-17T23:55:55.466733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:06:59.017416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.465258+0000","last_change":"2024-03-17T23:55:02.318944+0000","last_active":"2024-03-17T23:55:55.465258+0000","last_peered":"2024-03-17T23:55:55.465258+0000","last_clean":"2024-03-17T23:55:55.465258+0000","last_became_active":"2024-03-17T23:55:02.318432+0000","last_became_peered":"2024-03-17T23:55:02.318432+0000","last_unstale":"2024-03-17T23:55:55.465258+0000","last_undegraded":"2024-03-17T23:55:55.465258+0000","last_fullsized":"2024-03-17T23:55:55.465258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:23:47.986701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.483911+0000","last_change":"2024-03-17T23:54:55.873746+0000","last_active":"2024-03-17T23:55:55.483911+0000","last_peered":"2024-03-17T23:55:55.483911+0000","last_clean":"2024-03-17T23:55:55.483911+0000","last_became_active":"2024-03-17T23:54:55.872229+0000","last_became_peered":"2024-03-17T23:54:55.872229+0000","last_unstale":"2024-03-17T23:55:55.483911+0000","last_undegraded":"2024-03-17T23:55:55.483911+0000","last_fullsized":"2024-03-17T23:55:55.483911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:10.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.468358+0000","last_change":"2024-03-17T23:54:56.361294+0000","last_active":"2024-03-17T23:55:55.468358+0000","last_peered":"2024-03-17T23:55:55.468358+0000","last_clean":"2024-03-17T23:55:55.468358+0000","last_became_active":"2024-03-17T23:54:56.360976+0000","last_became_peered":"2024-03-17T23:54:56.360976+0000","last_unstale":"2024-03-17T23:55:55.468358+0000","last_undegraded":"2024-03-17T23:55:55.468358+0000","last_fullsized":"2024-03-17T23:55:55.468358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:28:58.685991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.475376+0000","last_change":"2024-03-17T23:55:01.880221+0000","last_active":"2024-03-17T23:55:55.475376+0000","last_peered":"2024-03-17T23:55:55.475376+0000","last_clean":"2024-03-17T23:55:55.475376+0000","last_became_active":"2024-03-17T23:55:01.879785+0000","last_became_peered":"2024-03-17T23:55:01.879785+0000","last_unstale":"2024-03-17T23:55:55.475376+0000","last_undegraded":"2024-03-17T23:55:55.475376+0000","last_fullsized":"2024-03-17T23:55:55.475376+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:24:51.698857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":116,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:47.165320+0000","last_change":"2024-03-17T23:54:55.854176+0000","last_active":"2024-03-17T23:56:47.165320+0000","last_peered":"2024-03-17T23:56:47.165320+0000","last_clean":"2024-03-17T23:56:47.165320+0000","last_became_active":"2024-03-17T23:54:55.853859+0000","last_became_peered":"2024-03-17T23:54:55.853859+0000","last_unstale":"2024-03-17T23:56:47.165320+0000","last_undegraded":"2024-03-17T23:56:47.165320+0000","last_fullsized":"2024-03-17T23:56:47.165320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:43:43.319635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.505626+0000","last_change":"2024-03-17T23:55:55.505626+0000","last_active":"2024-03-17T23:55:55.505626+0000","last_peered":"2024-03-17T23:55:55.505626+0000","last_clean":"2024-03-17T23:55:55.505626+0000","last_became_active":"2024-03-17T23:55:55.505284+0000","last_became_peered":"2024-03-17T23:55:55.505284+0000","last_unstale":"2024-03-17T23:55:55.505626+0000","last_undegraded":"2024-03-17T23:55:55.505626+0000","last_fullsized":"2024-03-17T23:55:55.505626+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:10:48.182303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.475236+0000","last_change":"2024-03-17T23:55:02.320494+0000","last_active":"2024-03-17T23:55:55.475236+0000","last_peered":"2024-03-17T23:55:55.475236+0000","last_clean":"2024-03-17T23:55:55.475236+0000","last_became_active":"2024-03-17T23:55:02.319188+0000","last_became_peered":"2024-03-17T23:55:02.319188+0000","last_unstale":"2024-03-17T23:55:55.475236+0000","last_undegraded":"2024-03-17T23:55:55.475236+0000","last_fullsized":"2024-03-17T23:55:55.475236+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:30:06.348718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470704+0000","last_change":"2024-03-17T23:54:56.340798+0000","last_active":"2024-03-17T23:55:55.470704+0000","last_peered":"2024-03-17T23:55:55.470704+0000","last_clean":"2024-03-17T23:55:55.470704+0000","last_became_active":"2024-03-17T23:54:56.340386+0000","last_became_peered":"2024-03-17T23:54:56.340386+0000","last_unstale":"2024-03-17T23:55:55.470704+0000","last_undegraded":"2024-03-17T23:55:55.470704+0000","last_fullsized":"2024-03-17T23:55:55.470704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:57:58.546464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":114,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:46.193085+0000","last_change":"2024-03-17T23:54:55.870475+0000","last_active":"2024-03-17T23:56:46.193085+0000","last_peered":"2024-03-17T23:56:46.193085+0000","last_clean":"2024-03-17T23:56:46.193085+0000","last_became_active":"2024-03-17T23:54:55.870119+0000","last_became_peered":"2024-03-17T23:54:55.870119+0000","last_unstale":"2024-03-17T23:56:46.193085+0000","last_undegraded":"2024-03-17T23:56:46.193085+0000","last_fullsized":"2024-03-17T23:56:46.193085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:12:15.421117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571442+0000","last_change":"2024-03-17T23:55:01.892759+0000","last_active":"2024-03-17T23:55:55.571442+0000","last_peered":"2024-03-17T23:55:55.571442+0000","last_clean":"2024-03-17T23:55:55.571442+0000","last_became_active":"2024-03-17T23:55:01.892240+0000","last_became_peered":"2024-03-17T23:55:01.892240+0000","last_unstale":"2024-03-17T23:55:55.571442+0000","last_undegraded":"2024-03-17T23:55:55.571442+0000","last_fullsized":"2024-03-17T23:55:55.571442+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:16:24.391971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.481738+0000","last_change":"2024-03-17T23:54:56.354898+0000","last_active":"2024-03-17T23:55:55.481738+0000","last_peered":"2024-03-17T23:55:55.481738+0000","last_clean":"2024-03-17T23:55:55.481738+0000","last_became_active":"2024-03-17T23:54:56.354345+0000","last_became_peered":"2024-03-17T23:54:56.354345+0000","last_unstale":"2024-03-17T23:55:55.481738+0000","last_undegraded":"2024-03-17T23:55:55.481738+0000","last_fullsized":"2024-03-17T23:55:55.481738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:18:56.962412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.469923+0000","last_change":"2024-03-17T23:54:55.846706+0000","last_active":"2024-03-17T23:55:55.469923+0000","last_peered":"2024-03-17T23:55:55.469923+0000","last_clean":"2024-03-17T23:55:55.469923+0000","last_became_active":"2024-03-17T23:54:55.846372+0000","last_became_peered":"2024-03-17T23:54:55.846372+0000","last_unstale":"2024-03-17T23:55:55.469923+0000","last_undegraded":"2024-03-17T23:55:55.469923+0000","last_fullsized":"2024-03-17T23:55:55.469923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:28:47.697975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":351,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.570430+0000","last_change":"2024-03-17T23:53:34.383413+0000","last_active":"2024-03-17T23:55:55.570430+0000","last_peered":"2024-03-17T23:55:55.570430+0000","last_clean":"2024-03-17T23:55:55.570430+0000","last_became_active":"2024-03-17T23:53:34.382875+0000","last_became_peered":"2024-03-17T23:53:34.382875+0000","last_unstale":"2024-03-17T23:55:55.570430+0000","last_undegraded":"2024-03-17T23:55:55.570430+0000","last_fullsized":"2024-03-17T23:55:55.570430+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:38.641018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:38.641018+0000","last_clean_scrub_stamp":"2024-03-17T23:49:38.641018+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:49:45.234628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.490220+0000","last_change":"2024-03-17T23:55:55.490220+0000","last_active":"2024-03-17T23:55:55.490220+0000","last_peered":"2024-03-17T23:55:55.490220+0000","last_clean":"2024-03-17T23:55:55.490220+0000","last_became_active":"2024-03-17T23:55:55.489804+0000","last_became_peered":"2024-03-17T23:55:55.489804+0000","last_unstale":"2024-03-17T23:55:55.490220+0000","last_undegraded":"2024-03-17T23:55:55.490220+0000","last_fullsized":"2024-03-17T23:55:55.490220+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:48.606538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.500027+0000","last_change":"2024-03-17T23:55:55.500027+0000","last_active":"2024-03-17T23:55:55.500027+0000","last_peered":"2024-03-17T23:55:55.500027+0000","last_clean":"2024-03-17T23:55:55.500027+0000","last_became_active":"2024-03-17T23:55:55.499624+0000","last_became_peered":"2024-03-17T23:55:55.499624+0000","last_unstale":"2024-03-17T23:55:55.500027+0000","last_undegraded":"2024-03-17T23:55:55.500027+0000","last_fullsized":"2024-03-17T23:55:55.500027+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:15:07.830593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.570448+0000","last_change":"2024-03-17T23:54:55.840743+0000","last_active":"2024-03-17T23:55:55.570448+0000","last_peered":"2024-03-17T23:55:55.570448+0000","last_clean":"2024-03-17T23:55:55.570448+0000","last_became_active":"2024-03-17T23:54:55.840463+0000","last_became_peered":"2024-03-17T23:54:55.840463+0000","last_unstale":"2024-03-17T23:55:55.570448+0000","last_undegraded":"2024-03-17T23:55:55.570448+0000","last_fullsized":"2024-03-17T23:55:55.570448+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:24:24.130911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466913+0000","last_change":"2024-03-17T23:55:01.876895+0000","last_active":"2024-03-17T23:55:55.466913+0000","last_peered":"2024-03-17T23:55:55.466913+0000","last_clean":"2024-03-17T23:55:55.466913+0000","last_became_active":"2024-03-17T23:55:01.876443+0000","last_became_peered":"2024-03-17T23:55:01.876443+0000","last_unstale":"2024-03-17T23:55:55.466913+0000","last_undegraded":"2024-03-17T23:55:55.466913+0000","last_fullsized":"2024-03-17T23:55:55.466913+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:51:55.737167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.517893+0000","last_change":"2024-03-17T23:55:55.517534+0000","last_active":"2024-03-17T23:55:55.517893+0000","last_peered":"2024-03-17T23:55:55.517893+0000","last_clean":"2024-03-17T23:55:55.517893+0000","last_became_active":"2024-03-17T23:55:55.499568+0000","last_became_peered":"2024-03-17T23:55:55.499568+0000","last_unstale":"2024-03-17T23:55:55.517893+0000","last_undegraded":"2024-03-17T23:55:55.517893+0000","last_fullsized":"2024-03-17T23:55:55.517893+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:36:23.064322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.475674+0000","last_change":"2024-03-17T23:54:56.365913+0000","last_active":"2024-03-17T23:55:55.475674+0000","last_peered":"2024-03-17T23:55:55.475674+0000","last_clean":"2024-03-17T23:55:55.475674+0000","last_became_active":"2024-03-17T23:54:56.365578+0000","last_became_peered":"2024-03-17T23:54:56.365578+0000","last_unstale":"2024-03-17T23:55:55.475674+0000","last_undegraded":"2024-03-17T23:55:55.475674+0000","last_fullsized":"2024-03-17T23:55:55.475674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:49.330026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.480298+0000","last_change":"2024-03-17T23:55:02.319060+0000","last_active":"2024-03-17T23:55:55.480298+0000","last_peered":"2024-03-17T23:55:55.480298+0000","last_clean":"2024-03-17T23:55:55.480298+0000","last_became_active":"2024-03-17T23:55:02.318542+0000","last_became_peered":"2024-03-17T23:55:02.318542+0000","last_unstale":"2024-03-17T23:55:55.480298+0000","last_undegraded":"2024-03-17T23:55:55.480298+0000","last_fullsized":"2024-03-17T23:55:55.480298+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:59:27.334039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'13","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.501025+0000","last_change":"2024-03-17T23:54:55.861011+0000","last_active":"2024-03-17T23:55:55.501025+0000","last_peered":"2024-03-17T23:55:55.501025+0000","last_clean":"2024-03-17T23:55:55.501025+0000","last_became_active":"2024-03-17T23:54:55.860388+0000","last_became_peered":"2024-03-17T23:54:55.860388+0000","last_unstale":"2024-03-17T23:55:55.501025+0000","last_undegraded":"2024-03-17T23:55:55.501025+0000","last_fullsized":"2024-03-17T23:55:55.501025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:20:11.943034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.483269+0000","last_change":"2024-03-17T23:54:56.357572+0000","last_active":"2024-03-17T23:55:55.483269+0000","last_peered":"2024-03-17T23:55:55.483269+0000","last_clean":"2024-03-17T23:55:55.483269+0000","last_became_active":"2024-03-17T23:54:56.357226+0000","last_became_peered":"2024-03-17T23:54:56.357226+0000","last_unstale":"2024-03-17T23:55:55.483269+0000","last_undegraded":"2024-03-17T23:55:55.483269+0000","last_fullsized":"2024-03-17T23:55:55.483269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:35:44.939454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466533+0000","last_change":"2024-03-17T23:55:01.860239+0000","last_active":"2024-03-17T23:55:55.466533+0000","last_peered":"2024-03-17T23:55:55.466533+0000","last_clean":"2024-03-17T23:55:55.466533+0000","last_became_active":"2024-03-17T23:55:01.859754+0000","last_became_peered":"2024-03-17T23:55:01.859754+0000","last_unstale":"2024-03-17T23:55:55.466533+0000","last_undegraded":"2024-03-17T23:55:55.466533+0000","last_fullsized":"2024-03-17T23:55:55.466533+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:30:16.833040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470570+0000","last_change":"2024-03-17T23:54:55.855917+0000","last_active":"2024-03-17T23:55:55.470570+0000","last_peered":"2024-03-17T23:55:55.470570+0000","last_clean":"2024-03-17T23:55:55.470570+0000","last_became_active":"2024-03-17T23:54:55.855562+0000","last_became_peered":"2024-03-17T23:54:55.855562+0000","last_unstale":"2024-03-17T23:55:55.470570+0000","last_undegraded":"2024-03-17T23:55:55.470570+0000","last_fullsized":"2024-03-17T23:55:55.470570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:44.928086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.493129+0000","last_change":"2024-03-17T23:54:56.362634+0000","last_active":"2024-03-17T23:55:55.493129+0000","last_peered":"2024-03-17T23:55:55.493129+0000","last_clean":"2024-03-17T23:55:55.493129+0000","last_became_active":"2024-03-17T23:54:56.361910+0000","last_became_peered":"2024-03-17T23:54:56.361910+0000","last_unstale":"2024-03-17T23:55:55.493129+0000","last_undegraded":"2024-03-17T23:55:55.493129+0000","last_fullsized":"2024-03-17T23:55:55.493129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:20:34.988074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.469836+0000","last_change":"2024-03-17T23:55:02.319090+0000","last_active":"2024-03-17T23:55:55.469836+0000","last_peered":"2024-03-17T23:55:55.469836+0000","last_clean":"2024-03-17T23:55:55.469836+0000","last_became_active":"2024-03-17T23:55:02.318695+0000","last_became_peered":"2024-03-17T23:55:02.318695+0000","last_unstale":"2024-03-17T23:55:55.469836+0000","last_undegraded":"2024-03-17T23:55:55.469836+0000","last_fullsized":"2024-03-17T23:55:55.469836+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:32:13.984797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.570814+0000","last_change":"2024-03-17T23:54:55.852999+0000","last_active":"2024-03-17T23:55:55.570814+0000","last_peered":"2024-03-17T23:55:55.570814+0000","last_clean":"2024-03-17T23:55:55.570814+0000","last_became_active":"2024-03-17T23:54:55.852653+0000","last_became_peered":"2024-03-17T23:54:55.852653+0000","last_unstale":"2024-03-17T23:55:55.570814+0000","last_undegraded":"2024-03-17T23:55:55.570814+0000","last_fullsized":"2024-03-17T23:55:55.570814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:04:10.561807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.464416+0000","last_change":"2024-03-17T23:54:56.345171+0000","last_active":"2024-03-17T23:55:55.464416+0000","last_peered":"2024-03-17T23:55:55.464416+0000","last_clean":"2024-03-17T23:55:55.464416+0000","last_became_active":"2024-03-17T23:54:56.344146+0000","last_became_peered":"2024-03-17T23:54:56.344146+0000","last_unstale":"2024-03-17T23:55:55.464416+0000","last_undegraded":"2024-03-17T23:55:55.464416+0000","last_fullsized":"2024-03-17T23:55:55.464416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:08.546622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.497287+0000","last_change":"2024-03-17T23:55:01.882655+0000","last_active":"2024-03-17T23:55:55.497287+0000","last_peered":"2024-03-17T23:55:55.497287+0000","last_clean":"2024-03-17T23:55:55.497287+0000","last_became_active":"2024-03-17T23:55:01.882147+0000","last_became_peered":"2024-03-17T23:55:01.882147+0000","last_unstale":"2024-03-17T23:55:55.497287+0000","last_undegraded":"2024-03-17T23:55:55.497287+0000","last_fullsized":"2024-03-17T23:55:55.497287+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:17:55.653103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.480958+0000","last_change":"2024-03-17T23:54:55.862499+0000","last_active":"2024-03-17T23:55:55.480958+0000","last_peered":"2024-03-17T23:55:55.480958+0000","last_clean":"2024-03-17T23:55:55.480958+0000","last_became_active":"2024-03-17T23:54:55.862003+0000","last_became_peered":"2024-03-17T23:54:55.862003+0000","last_unstale":"2024-03-17T23:55:55.480958+0000","last_undegraded":"2024-03-17T23:55:55.480958+0000","last_fullsized":"2024-03-17T23:55:55.480958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:37:19.046543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466673+0000","last_change":"2024-03-17T23:54:56.352154+0000","last_active":"2024-03-17T23:55:55.466673+0000","last_peered":"2024-03-17T23:55:55.466673+0000","last_clean":"2024-03-17T23:55:55.466673+0000","last_became_active":"2024-03-17T23:54:56.351818+0000","last_became_peered":"2024-03-17T23:54:56.351818+0000","last_unstale":"2024-03-17T23:55:55.466673+0000","last_undegraded":"2024-03-17T23:55:55.466673+0000","last_fullsized":"2024-03-17T23:55:55.466673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:44:46.538778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.469859+0000","last_change":"2024-03-17T23:55:01.876605+0000","last_active":"2024-03-17T23:55:55.469859+0000","last_peered":"2024-03-17T23:55:55.469859+0000","last_clean":"2024-03-17T23:55:55.469859+0000","last_became_active":"2024-03-17T23:55:01.876253+0000","last_became_peered":"2024-03-17T23:55:01.876253+0000","last_unstale":"2024-03-17T23:55:55.469859+0000","last_undegraded":"2024-03-17T23:55:55.469859+0000","last_fullsized":"2024-03-17T23:55:55.469859+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:27:43.271798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466918+0000","last_change":"2024-03-17T23:54:55.862209+0000","last_active":"2024-03-17T23:55:55.466918+0000","last_peered":"2024-03-17T23:55:55.466918+0000","last_clean":"2024-03-17T23:55:55.466918+0000","last_became_active":"2024-03-17T23:54:55.861829+0000","last_became_peered":"2024-03-17T23:54:55.861829+0000","last_unstale":"2024-03-17T23:55:55.466918+0000","last_undegraded":"2024-03-17T23:55:55.466918+0000","last_fullsized":"2024-03-17T23:55:55.466918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:13:58.756538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.479975+0000","last_change":"2024-03-17T23:54:56.366869+0000","last_active":"2024-03-17T23:55:55.479975+0000","last_peered":"2024-03-17T23:55:55.479975+0000","last_clean":"2024-03-17T23:55:55.479975+0000","last_became_active":"2024-03-17T23:54:56.365380+0000","last_became_peered":"2024-03-17T23:54:56.365380+0000","last_unstale":"2024-03-17T23:55:55.479975+0000","last_undegraded":"2024-03-17T23:55:55.479975+0000","last_fullsized":"2024-03-17T23:55:55.479975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:28:03.010849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.469920+0000","last_change":"2024-03-17T23:55:02.318820+0000","last_active":"2024-03-17T23:55:55.469920+0000","last_peered":"2024-03-17T23:55:55.469920+0000","last_clean":"2024-03-17T23:55:55.469920+0000","last_became_active":"2024-03-17T23:55:02.318308+0000","last_became_peered":"2024-03-17T23:55:02.318308+0000","last_unstale":"2024-03-17T23:55:55.469920+0000","last_undegraded":"2024-03-17T23:55:55.469920+0000","last_fullsized":"2024-03-17T23:55:55.469920+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:40:35.224105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.464575+0000","last_change":"2024-03-17T23:54:55.837980+0000","last_active":"2024-03-17T23:55:55.464575+0000","last_peered":"2024-03-17T23:55:55.464575+0000","last_clean":"2024-03-17T23:55:55.464575+0000","last_became_active":"2024-03-17T23:54:55.837614+0000","last_became_peered":"2024-03-17T23:54:55.837614+0000","last_unstale":"2024-03-17T23:55:55.464575+0000","last_undegraded":"2024-03-17T23:55:55.464575+0000","last_fullsized":"2024-03-17T23:55:55.464575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:05:07.228281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571194+0000","last_change":"2024-03-17T23:54:56.348534+0000","last_active":"2024-03-17T23:55:55.571194+0000","last_peered":"2024-03-17T23:55:55.571194+0000","last_clean":"2024-03-17T23:55:55.571194+0000","last_became_active":"2024-03-17T23:54:56.348138+0000","last_became_peered":"2024-03-17T23:54:56.348138+0000","last_unstale":"2024-03-17T23:55:55.571194+0000","last_undegraded":"2024-03-17T23:55:55.571194+0000","last_fullsized":"2024-03-17T23:55:55.571194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:47:49.481608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571122+0000","last_change":"2024-03-17T23:55:01.876287+0000","last_active":"2024-03-17T23:55:55.571122+0000","last_peered":"2024-03-17T23:55:55.571122+0000","last_clean":"2024-03-17T23:55:55.571122+0000","last_became_active":"2024-03-17T23:55:01.875861+0000","last_became_peered":"2024-03-17T23:55:01.875861+0000","last_unstale":"2024-03-17T23:55:55.571122+0000","last_undegraded":"2024-03-17T23:55:55.571122+0000","last_fullsized":"2024-03-17T23:55:55.571122+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:33:59.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":117,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:47.123755+0000","last_change":"2024-03-17T23:54:55.870989+0000","last_active":"2024-03-17T23:56:47.123755+0000","last_peered":"2024-03-17T23:56:47.123755+0000","last_clean":"2024-03-17T23:56:47.123755+0000","last_became_active":"2024-03-17T23:54:55.870633+0000","last_became_peered":"2024-03-17T23:54:55.870633+0000","last_unstale":"2024-03-17T23:56:47.123755+0000","last_undegraded":"2024-03-17T23:56:47.123755+0000","last_fullsized":"2024-03-17T23:56:47.123755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:55.310289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571567+0000","last_change":"2024-03-17T23:54:56.345874+0000","last_active":"2024-03-17T23:55:55.571567+0000","last_peered":"2024-03-17T23:55:55.571567+0000","last_clean":"2024-03-17T23:55:55.571567+0000","last_became_active":"2024-03-17T23:54:56.345547+0000","last_became_peered":"2024-03-17T23:54:56.345547+0000","last_unstale":"2024-03-17T23:55:55.571567+0000","last_undegraded":"2024-03-17T23:55:55.571567+0000","last_fullsized":"2024-03-17T23:55:55.571567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:52:31.724224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.484602+0000","last_change":"2024-03-17T23:55:01.892947+0000","last_active":"2024-03-17T23:55:55.484602+0000","last_peered":"2024-03-17T23:55:55.484602+0000","last_clean":"2024-03-17T23:55:55.484602+0000","last_became_active":"2024-03-17T23:55:01.892481+0000","last_became_peered":"2024-03-17T23:55:01.892481+0000","last_unstale":"2024-03-17T23:55:55.484602+0000","last_undegraded":"2024-03-17T23:55:55.484602+0000","last_fullsized":"2024-03-17T23:55:55.484602+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:07:14.901490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.471113+0000","last_change":"2024-03-17T23:54:55.857943+0000","last_active":"2024-03-17T23:55:55.471113+0000","last_peered":"2024-03-17T23:55:55.471113+0000","last_clean":"2024-03-17T23:55:55.471113+0000","last_became_active":"2024-03-17T23:54:55.857592+0000","last_became_peered":"2024-03-17T23:54:55.857592+0000","last_unstale":"2024-03-17T23:55:55.471113+0000","last_undegraded":"2024-03-17T23:55:55.471113+0000","last_fullsized":"2024-03-17T23:55:55.471113+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:50:03.643518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571123+0000","last_change":"2024-03-17T23:54:56.357496+0000","last_active":"2024-03-17T23:55:55.571123+0000","last_peered":"2024-03-17T23:55:55.571123+0000","last_clean":"2024-03-17T23:55:55.571123+0000","last_became_active":"2024-03-17T23:54:56.357242+0000","last_became_peered":"2024-03-17T23:54:56.357242+0000","last_unstale":"2024-03-17T23:55:55.571123+0000","last_undegraded":"2024-03-17T23:55:55.571123+0000","last_fullsized":"2024-03-17T23:55:55.571123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:52:49.941623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470176+0000","last_change":"2024-03-17T23:55:01.866288+0000","last_active":"2024-03-17T23:55:55.470176+0000","last_peered":"2024-03-17T23:55:55.470176+0000","last_clean":"2024-03-17T23:55:55.470176+0000","last_became_active":"2024-03-17T23:55:01.865875+0000","last_became_peered":"2024-03-17T23:55:01.865875+0000","last_unstale":"2024-03-17T23:55:55.470176+0000","last_undegraded":"2024-03-17T23:55:55.470176+0000","last_fullsized":"2024-03-17T23:55:55.470176+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:45:43.198830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.476761+0000","last_change":"2024-03-17T23:54:55.871971+0000","last_active":"2024-03-17T23:55:55.476761+0000","last_peered":"2024-03-17T23:55:55.476761+0000","last_clean":"2024-03-17T23:55:55.476761+0000","last_became_active":"2024-03-17T23:54:55.871434+0000","last_became_peered":"2024-03-17T23:54:55.871434+0000","last_unstale":"2024-03-17T23:55:55.476761+0000","last_undegraded":"2024-03-17T23:55:55.476761+0000","last_fullsized":"2024-03-17T23:55:55.476761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:41:10.667628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571377+0000","last_change":"2024-03-17T23:54:56.343595+0000","last_active":"2024-03-17T23:55:55.571377+0000","last_peered":"2024-03-17T23:55:55.571377+0000","last_clean":"2024-03-17T23:55:55.571377+0000","last_became_active":"2024-03-17T23:54:56.343270+0000","last_became_peered":"2024-03-17T23:54:56.343270+0000","last_unstale":"2024-03-17T23:55:55.571377+0000","last_undegraded":"2024-03-17T23:55:55.571377+0000","last_fullsized":"2024-03-17T23:55:55.571377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:22:05.904023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.467028+0000","last_change":"2024-03-17T23:54:56.346571+0000","last_active":"2024-03-17T23:55:55.467028+0000","last_peered":"2024-03-17T23:55:55.467028+0000","last_clean":"2024-03-17T23:55:55.467028+0000","last_became_active":"2024-03-17T23:54:56.346234+0000","last_became_peered":"2024-03-17T23:54:56.346234+0000","last_unstale":"2024-03-17T23:55:55.467028+0000","last_undegraded":"2024-03-17T23:55:55.467028+0000","last_fullsized":"2024-03-17T23:55:55.467028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:25.448506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466928+0000","last_change":"2024-03-17T23:54:56.342299+0000","last_active":"2024-03-17T23:55:55.466928+0000","last_peered":"2024-03-17T23:55:55.466928+0000","last_clean":"2024-03-17T23:55:55.466928+0000","last_became_active":"2024-03-17T23:54:56.341988+0000","last_became_peered":"2024-03-17T23:54:56.341988+0000","last_unstale":"2024-03-17T23:55:55.466928+0000","last_undegraded":"2024-03-17T23:55:55.466928+0000","last_fullsized":"2024-03-17T23:55:55.466928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:03:38.595595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466805+0000","last_change":"2024-03-17T23:54:56.360601+0000","last_active":"2024-03-17T23:55:55.466805+0000","last_peered":"2024-03-17T23:55:55.466805+0000","last_clean":"2024-03-17T23:55:55.466805+0000","last_became_active":"2024-03-17T23:54:56.360215+0000","last_became_peered":"2024-03-17T23:54:56.360215+0000","last_unstale":"2024-03-17T23:55:55.466805+0000","last_undegraded":"2024-03-17T23:55:55.466805+0000","last_fullsized":"2024-03-17T23:55:55.466805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:42:07.661460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.484863+0000","last_change":"2024-03-17T23:54:56.358523+0000","last_active":"2024-03-17T23:55:55.484863+0000","last_peered":"2024-03-17T23:55:55.484863+0000","last_clean":"2024-03-17T23:55:55.484863+0000","last_became_active":"2024-03-17T23:54:56.358172+0000","last_became_peered":"2024-03-17T23:54:56.358172+0000","last_unstale":"2024-03-17T23:55:55.484863+0000","last_undegraded":"2024-03-17T23:55:55.484863+0000","last_fullsized":"2024-03-17T23:55:55.484863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:00.542176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.475571+0000","last_change":"2024-03-17T23:54:56.348619+0000","last_active":"2024-03-17T23:55:55.475571+0000","last_peered":"2024-03-17T23:55:55.475571+0000","last_clean":"2024-03-17T23:55:55.475571+0000","last_became_active":"2024-03-17T23:54:56.348309+0000","last_became_peered":"2024-03-17T23:54:56.348309+0000","last_unstale":"2024-03-17T23:55:55.475571+0000","last_undegraded":"2024-03-17T23:55:55.475571+0000","last_fullsized":"2024-03-17T23:55:55.475571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:38:42.130689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.468378+0000","last_change":"2024-03-17T23:54:56.347736+0000","last_active":"2024-03-17T23:55:55.468378+0000","last_peered":"2024-03-17T23:55:55.468378+0000","last_clean":"2024-03-17T23:55:55.468378+0000","last_became_active":"2024-03-17T23:54:56.347283+0000","last_became_peered":"2024-03-17T23:54:56.347283+0000","last_unstale":"2024-03-17T23:55:55.468378+0000","last_undegraded":"2024-03-17T23:55:55.468378+0000","last_fullsized":"2024-03-17T23:55:55.468378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:35:00.683852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.468285+0000","last_change":"2024-03-17T23:54:56.344574+0000","last_active":"2024-03-17T23:55:55.468285+0000","last_peered":"2024-03-17T23:55:55.468285+0000","last_clean":"2024-03-17T23:55:55.468285+0000","last_became_active":"2024-03-17T23:54:56.344272+0000","last_became_peered":"2024-03-17T23:54:56.344272+0000","last_unstale":"2024-03-17T23:55:55.468285+0000","last_undegraded":"2024-03-17T23:55:55.468285+0000","last_fullsized":"2024-03-17T23:55:55.468285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:37:37.134152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.483462+0000","last_change":"2024-03-17T23:54:56.365633+0000","last_active":"2024-03-17T23:55:55.483462+0000","last_peered":"2024-03-17T23:55:55.483462+0000","last_clean":"2024-03-17T23:55:55.483462+0000","last_became_active":"2024-03-17T23:54:56.364533+0000","last_became_peered":"2024-03-17T23:54:56.364533+0000","last_unstale":"2024-03-17T23:55:55.483462+0000","last_undegraded":"2024-03-17T23:55:55.483462+0000","last_fullsized":"2024-03-17T23:55:55.483462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:05:20.848672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470661+0000","last_change":"2024-03-17T23:54:56.365614+0000","last_active":"2024-03-17T23:55:55.470661+0000","last_peered":"2024-03-17T23:55:55.470661+0000","last_clean":"2024-03-17T23:55:55.470661+0000","last_became_active":"2024-03-17T23:54:56.365301+0000","last_became_peered":"2024-03-17T23:54:56.365301+0000","last_unstale":"2024-03-17T23:55:55.470661+0000","last_undegraded":"2024-03-17T23:55:55.470661+0000","last_fullsized":"2024-03-17T23:55:55.470661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:20.122964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.481102+0000","last_change":"2024-03-17T23:54:56.365531+0000","last_active":"2024-03-17T23:55:55.481102+0000","last_peered":"2024-03-17T23:55:55.481102+0000","last_clean":"2024-03-17T23:55:55.481102+0000","last_became_active":"2024-03-17T23:54:56.364675+0000","last_became_peered":"2024-03-17T23:54:56.364675+0000","last_unstale":"2024-03-17T23:55:55.481102+0000","last_undegraded":"2024-03-17T23:55:55.481102+0000","last_fullsized":"2024-03-17T23:55:55.481102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:25:49.612055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571525+0000","last_change":"2024-03-17T23:54:56.355306+0000","last_active":"2024-03-17T23:55:55.571525+0000","last_peered":"2024-03-17T23:55:55.571525+0000","last_clean":"2024-03-17T23:55:55.571525+0000","last_became_active":"2024-03-17T23:54:56.354975+0000","last_became_peered":"2024-03-17T23:54:56.354975+0000","last_unstale":"2024-03-17T23:55:55.571525+0000","last_undegraded":"2024-03-17T23:55:55.571525+0000","last_fullsized":"2024-03-17T23:55:55.571525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:58.606691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.477810+0000","last_change":"2024-03-17T23:54:56.350333+0000","last_active":"2024-03-17T23:55:55.477810+0000","last_peered":"2024-03-17T23:55:55.477810+0000","last_clean":"2024-03-17T23:55:55.477810+0000","last_became_active":"2024-03-17T23:54:56.349953+0000","last_became_peered":"2024-03-17T23:54:56.349953+0000","last_unstale":"2024-03-17T23:55:55.477810+0000","last_undegraded":"2024-03-17T23:55:55.477810+0000","last_fullsized":"2024-03-17T23:55:55.477810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:19:34.639852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466946+0000","last_change":"2024-03-17T23:54:56.343143+0000","last_active":"2024-03-17T23:55:55.466946+0000","last_peered":"2024-03-17T23:55:55.466946+0000","last_clean":"2024-03-17T23:55:55.466946+0000","last_became_active":"2024-03-17T23:54:56.342772+0000","last_became_peered":"2024-03-17T23:54:56.342772+0000","last_unstale":"2024-03-17T23:55:55.466946+0000","last_undegraded":"2024-03-17T23:55:55.466946+0000","last_fullsized":"2024-03-17T23:55:55.466946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:33:45.668975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246647,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71019,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":854,"num_read_kb":506,"num_write":243,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":292,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":213057,"data_compressed":5505,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":509792,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1163264,"data_stored":1160634,"data_compressed":58192,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":513638,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":509808,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":1106417,"data_compressed":56357,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":505046,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":570282,"data_compressed":22487,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":506502,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":570713,"data_compressed":22487,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":504754,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":509232,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":1101999,"data_compressed":56357,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34061,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34061,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":65958,"data_compressed":1835,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66389,"data_compressed":1835,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":65942,"data_compressed":1835,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:56:53.355 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-17T23:56:53.834 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:53.977 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:53 smithi012 ceph-mon[31541]: pgmap v381: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:56:53.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:53 smithi012 ceph-mon[31541]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:56:54.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:53 smithi079 ceph-mon[36241]: pgmap v381: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:56:54.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:53 smithi079 ceph-mon[36241]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:56:54.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:53 smithi191 ceph-mon[36708]: pgmap v381: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:56:54.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:53 smithi191 ceph-mon[36708]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:56:55.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-03-17T23:56:55.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-03-17T23:56:55.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-03-17T23:56:55.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:55 smithi012 ceph-mon[31541]: pgmap v382: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:55.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-03-17T23:56:55.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:55 smithi012 ceph-mon[31541]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-17T23:56:55.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:55 smithi012 ceph-mon[31541]: osdmap e93: 12 total, 12 up, 12 in 2024-03-17T23:56:56.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:55 smithi079 ceph-mon[36241]: pgmap v382: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:56.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-03-17T23:56:56.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:55 smithi079 ceph-mon[36241]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-17T23:56:56.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:55 smithi079 ceph-mon[36241]: osdmap e93: 12 total, 12 up, 12 in 2024-03-17T23:56:56.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:55 smithi191 ceph-mon[36708]: pgmap v382: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:56:56.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-03-17T23:56:56.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:55 smithi191 ceph-mon[36708]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-17T23:56:56.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:55 smithi191 ceph-mon[36708]: osdmap e93: 12 total, 12 up, 12 in 2024-03-17T23:56:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.136+0000 7f17a8a00700 1 -- 172.21.15.12:0/2515297768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17a4102410 msgr2=0x7f17a41027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.136+0000 7f17a8a00700 1 --2- 172.21.15.12:0/2515297768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17a4102410 0x7f17a41027f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f17940099c0 tx=0x7f1794009cd0 comp rx=0 tx=0).stop 2024-03-17T23:56:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.137+0000 7f17a8a00700 1 -- 172.21.15.12:0/2515297768 shutdown_connections 2024-03-17T23:56:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.137+0000 7f17a8a00700 1 --2- 172.21.15.12:0/2515297768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a41074f0 0x7f17a410f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.137+0000 7f17a8a00700 1 --2- 172.21.15.12:0/2515297768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17a4102410 0x7f17a41027f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.137+0000 7f17a8a00700 1 --2- 172.21.15.12:0/2515297768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17a4102dc0 0x7f17a4106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.137+0000 7f17a8a00700 1 -- 172.21.15.12:0/2515297768 >> 172.21.15.12:0/2515297768 conn(0x7f17a40fdca0 msgr2=0x7f17a41000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.137+0000 7f17a8a00700 1 -- 172.21.15.12:0/2515297768 shutdown_connections 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.137+0000 7f17a8a00700 1 -- 172.21.15.12:0/2515297768 wait complete. 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.138+0000 7f17a8a00700 1 Processor -- start 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.138+0000 7f17a8a00700 1 -- start start 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.138+0000 7f17a8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a4102410 0x7f17a41a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.138+0000 7f17a8a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17a4102dc0 0x7f17a41a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.138+0000 7f17a8a00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17a41074f0 0x7f17a419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.138+0000 7f17a8a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f17a4112540 con 0x7f17a4102410 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.138+0000 7f17a8a00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41123c0 con 0x7f17a41074f0 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.138+0000 7f17a8a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41126c0 con 0x7f17a4102dc0 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.139+0000 7f17a2a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17a4102dc0 0x7f17a41a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.139+0000 7f17a3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a4102410 0x7f17a41a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.139+0000 7f17a2a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17a4102dc0 0x7f17a41a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:36888/0 (socket says 172.21.15.12:36888) 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.139+0000 7f17a2a00700 1 -- 172.21.15.12:0/289581254 learned_addr learned my addr 172.21.15.12:0/289581254 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.139+0000 7f17a3e00700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17a41074f0 0x7f17a419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.139+0000 7f17a2a00700 1 -- 172.21.15.12:0/289581254 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17a41074f0 msgr2=0x7f17a419a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.139+0000 7f17a2a00700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17a41074f0 0x7f17a419a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.139+0000 7f17a2a00700 1 -- 172.21.15.12:0/289581254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a4102410 msgr2=0x7f17a41a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.139+0000 7f17a2a00700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a4102410 0x7f17a41a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.139+0000 7f17a2a00700 1 -- 172.21.15.12:0/289581254 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a419b140 con 0x7f17a4102dc0 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.139+0000 7f17a3400700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a4102410 0x7f17a41a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.139+0000 7f17a3e00700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17a41074f0 0x7f17a419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.139+0000 7f17a2a00700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17a4102dc0 0x7f17a41a0d60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f179000e9f0 tx=0x7f179000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.140+0000 7f179be00700 1 -- 172.21.15.12:0/289581254 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1790004a30 con 0x7f17a4102dc0 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.140+0000 7f179be00700 1 -- 172.21.15.12:0/289581254 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f179000cca0 con 0x7f17a4102dc0 2024-03-17T23:56:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.140+0000 7f17a8a00700 1 -- 172.21.15.12:0/289581254 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a419b430 con 0x7f17a4102dc0 2024-03-17T23:56:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.140+0000 7f179be00700 1 -- 172.21.15.12:0/289581254 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17900056c0 con 0x7f17a4102dc0 2024-03-17T23:56:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.140+0000 7f17a8a00700 1 -- 172.21.15.12:0/289581254 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a41a76f0 con 0x7f17a4102dc0 2024-03-17T23:56:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.141+0000 7f17a8a00700 1 -- 172.21.15.12:0/289581254 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17a4068f90 con 0x7f17a4102dc0 2024-03-17T23:56:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.145+0000 7f179be00700 1 -- 172.21.15.12:0/289581254 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f179000c7c0 con 0x7f17a4102dc0 2024-03-17T23:56:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.146+0000 7f179be00700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f17880a2910 0x7f17880a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:56:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.146+0000 7f179be00700 1 -- 172.21.15.12:0/289581254 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f17900cf1f0 con 0x7f17a4102dc0 2024-03-17T23:56:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.147+0000 7f17a3400700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f17880a2910 0x7f17880a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:56:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.147+0000 7f17a3400700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f17880a2910 0x7f17880a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1794000c00 tx=0x7f1794005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:56:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.151+0000 7f179be00700 1 -- 172.21.15.12:0/289581254 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1790053740 con 0x7f17a4102dc0 2024-03-17T23:56:56.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.382+0000 7f17a8a00700 1 -- 172.21.15.12:0/289581254 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f17a410eaa0 con 0x7f17880a2910 2024-03-17T23:56:56.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.404+0000 7f179be00700 1 -- 172.21.15.12:0/289581254 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340848 (secure 0 0 0) 0x7f17a410eaa0 con 0x7f17880a2910 2024-03-17T23:56:56.413 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:56:56.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.412+0000 7f17a8a00700 1 -- 172.21.15.12:0/289581254 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f17880a2910 msgr2=0x7f17880a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:56.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.412+0000 7f17a8a00700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f17880a2910 0x7f17880a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1794000c00 tx=0x7f1794005fb0 comp rx=0 tx=0).stop 2024-03-17T23:56:56.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.412+0000 7f17a8a00700 1 -- 172.21.15.12:0/289581254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17a4102dc0 msgr2=0x7f17a41a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:56:56.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.412+0000 7f17a8a00700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17a4102dc0 0x7f17a41a0d60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f179000e9f0 tx=0x7f179000ed00 comp rx=0 tx=0).stop 2024-03-17T23:56:56.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.413+0000 7f17a8a00700 1 -- 172.21.15.12:0/289581254 shutdown_connections 2024-03-17T23:56:56.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.413+0000 7f17a8a00700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f17880a2910 0x7f17880a4dd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:56.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.413+0000 7f17a8a00700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17a41074f0 0x7f17a419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:56.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.413+0000 7f17a8a00700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17a4102410 0x7f17a41a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:56.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.413+0000 7f17a8a00700 1 --2- 172.21.15.12:0/289581254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17a4102dc0 0x7f17a41a0d60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:56:56.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.413+0000 7f17a8a00700 1 -- 172.21.15.12:0/289581254 >> 172.21.15.12:0/289581254 conn(0x7f17a40fdca0 msgr2=0x7f17a410e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:56:56.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.413+0000 7f17a8a00700 1 -- 172.21.15.12:0/289581254 shutdown_connections 2024-03-17T23:56:56.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:56:56.413+0000 7f17a8a00700 1 -- 172.21.15.12:0/289581254 wait complete. 2024-03-17T23:56:56.417 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-17T23:56:57.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:56 smithi079 ceph-mon[36241]: osdmap e94: 12 total, 12 up, 12 in 2024-03-17T23:56:57.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:56 smithi191 ceph-mon[36708]: osdmap e94: 12 total, 12 up, 12 in 2024-03-17T23:56:57.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:56 smithi012 ceph-mon[31541]: osdmap e94: 12 total, 12 up, 12 in 2024-03-17T23:56:57.298 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-03-17T23:56:56.148223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661870,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1032,"num_read_kb":642,"num_write":432,"num_write_kb":3085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1181028,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":557,"ondisk_log_size":557,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353328,"kb_used_data":12440,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588496,"statfs":{"total":1151940427776,"available":1151578619904,"internally_reserved":0,"allocated":12738560,"data_stored":8069315,"data_compressed":360444,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":47,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003218"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.467926+0000","last_change":"2024-03-17T23:54:56.354062+0000","last_active":"2024-03-17T23:55:55.467926+0000","last_peered":"2024-03-17T23:55:55.467926+0000","last_clean":"2024-03-17T23:55:55.467926+0000","last_became_active":"2024-03-17T23:54:56.353753+0000","last_became_peered":"2024-03-17T23:54:56.353753+0000","last_unstale":"2024-03-17T23:55:55.467926+0000","last_undegraded":"2024-03-17T23:55:55.467926+0000","last_fullsized":"2024-03-17T23:55:55.467926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:41:50.385420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.112173+0000","last_change":"2024-03-17T23:54:56.356507+0000","last_active":"2024-03-17T23:56:54.112173+0000","last_peered":"2024-03-17T23:56:54.112173+0000","last_clean":"2024-03-17T23:56:54.112173+0000","last_became_active":"2024-03-17T23:54:56.355714+0000","last_became_peered":"2024-03-17T23:54:56.355714+0000","last_unstale":"2024-03-17T23:56:54.112173+0000","last_undegraded":"2024-03-17T23:56:54.112173+0000","last_fullsized":"2024-03-17T23:56:54.112173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:44:15.992630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.127145+0000","last_change":"2024-03-17T23:54:56.352207+0000","last_active":"2024-03-17T23:56:54.127145+0000","last_peered":"2024-03-17T23:56:54.127145+0000","last_clean":"2024-03-17T23:56:54.127145+0000","last_became_active":"2024-03-17T23:54:56.351704+0000","last_became_peered":"2024-03-17T23:54:56.351704+0000","last_unstale":"2024-03-17T23:56:54.127145+0000","last_undegraded":"2024-03-17T23:56:54.127145+0000","last_fullsized":"2024-03-17T23:56:54.127145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:34.328120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.095756+0000","last_change":"2024-03-17T23:54:56.343336+0000","last_active":"2024-03-17T23:56:54.095756+0000","last_peered":"2024-03-17T23:56:54.095756+0000","last_clean":"2024-03-17T23:56:54.095756+0000","last_became_active":"2024-03-17T23:54:56.343033+0000","last_became_peered":"2024-03-17T23:54:56.343033+0000","last_unstale":"2024-03-17T23:56:54.095756+0000","last_undegraded":"2024-03-17T23:56:54.095756+0000","last_fullsized":"2024-03-17T23:56:54.095756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:40:21.814558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.118124+0000","last_change":"2024-03-17T23:54:56.360541+0000","last_active":"2024-03-17T23:56:54.118124+0000","last_peered":"2024-03-17T23:56:54.118124+0000","last_clean":"2024-03-17T23:56:54.118124+0000","last_became_active":"2024-03-17T23:54:56.360204+0000","last_became_peered":"2024-03-17T23:54:56.360204+0000","last_unstale":"2024-03-17T23:56:54.118124+0000","last_undegraded":"2024-03-17T23:56:54.118124+0000","last_fullsized":"2024-03-17T23:56:54.118124+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:57:37.853168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:55.132942+0000","last_change":"2024-03-17T23:54:56.363727+0000","last_active":"2024-03-17T23:56:55.132942+0000","last_peered":"2024-03-17T23:56:55.132942+0000","last_clean":"2024-03-17T23:56:55.132942+0000","last_became_active":"2024-03-17T23:54:56.363399+0000","last_became_peered":"2024-03-17T23:54:56.363399+0000","last_unstale":"2024-03-17T23:56:55.132942+0000","last_undegraded":"2024-03-17T23:56:55.132942+0000","last_fullsized":"2024-03-17T23:56:55.132942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:42.451379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.570468+0000","last_change":"2024-03-17T23:54:56.361881+0000","last_active":"2024-03-17T23:55:55.570468+0000","last_peered":"2024-03-17T23:55:55.570468+0000","last_clean":"2024-03-17T23:55:55.570468+0000","last_became_active":"2024-03-17T23:54:56.361531+0000","last_became_peered":"2024-03-17T23:54:56.361531+0000","last_unstale":"2024-03-17T23:55:55.570468+0000","last_undegraded":"2024-03-17T23:55:55.570468+0000","last_fullsized":"2024-03-17T23:55:55.570468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:44:39.792763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.480556+0000","last_change":"2024-03-17T23:54:56.358429+0000","last_active":"2024-03-17T23:55:55.480556+0000","last_peered":"2024-03-17T23:55:55.480556+0000","last_clean":"2024-03-17T23:55:55.480556+0000","last_became_active":"2024-03-17T23:54:56.355268+0000","last_became_peered":"2024-03-17T23:54:56.355268+0000","last_unstale":"2024-03-17T23:55:55.480556+0000","last_undegraded":"2024-03-17T23:55:55.480556+0000","last_fullsized":"2024-03-17T23:55:55.480556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:00:37.687542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.107891+0000","last_change":"2024-03-17T23:54:56.344228+0000","last_active":"2024-03-17T23:56:54.107891+0000","last_peered":"2024-03-17T23:56:54.107891+0000","last_clean":"2024-03-17T23:56:54.107891+0000","last_became_active":"2024-03-17T23:54:56.343863+0000","last_became_peered":"2024-03-17T23:54:56.343863+0000","last_unstale":"2024-03-17T23:56:54.107891+0000","last_undegraded":"2024-03-17T23:56:54.107891+0000","last_fullsized":"2024-03-17T23:56:54.107891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:45:12.552611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:55.133287+0000","last_change":"2024-03-17T23:54:56.341215+0000","last_active":"2024-03-17T23:56:55.133287+0000","last_peered":"2024-03-17T23:56:55.133287+0000","last_clean":"2024-03-17T23:56:55.133287+0000","last_became_active":"2024-03-17T23:54:56.340850+0000","last_became_peered":"2024-03-17T23:54:56.340850+0000","last_unstale":"2024-03-17T23:56:55.133287+0000","last_undegraded":"2024-03-17T23:56:55.133287+0000","last_fullsized":"2024-03-17T23:56:55.133287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:28.273320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571564+0000","last_change":"2024-03-17T23:54:56.346613+0000","last_active":"2024-03-17T23:55:55.571564+0000","last_peered":"2024-03-17T23:55:55.571564+0000","last_clean":"2024-03-17T23:55:55.571564+0000","last_became_active":"2024-03-17T23:54:56.345907+0000","last_became_peered":"2024-03-17T23:54:56.345907+0000","last_unstale":"2024-03-17T23:55:55.571564+0000","last_undegraded":"2024-03-17T23:55:55.571564+0000","last_fullsized":"2024-03-17T23:55:55.571564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:21.493841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.095972+0000","last_change":"2024-03-17T23:54:56.341924+0000","last_active":"2024-03-17T23:56:54.095972+0000","last_peered":"2024-03-17T23:56:54.095972+0000","last_clean":"2024-03-17T23:56:54.095972+0000","last_became_active":"2024-03-17T23:54:56.341469+0000","last_became_peered":"2024-03-17T23:54:56.341469+0000","last_unstale":"2024-03-17T23:56:54.095972+0000","last_undegraded":"2024-03-17T23:56:54.095972+0000","last_fullsized":"2024-03-17T23:56:54.095972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:49:47.256954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.477694+0000","last_change":"2024-03-17T23:54:56.332469+0000","last_active":"2024-03-17T23:55:55.477694+0000","last_peered":"2024-03-17T23:55:55.477694+0000","last_clean":"2024-03-17T23:55:55.477694+0000","last_became_active":"2024-03-17T23:54:56.330684+0000","last_became_peered":"2024-03-17T23:54:56.330684+0000","last_unstale":"2024-03-17T23:55:55.477694+0000","last_undegraded":"2024-03-17T23:55:55.477694+0000","last_fullsized":"2024-03-17T23:55:55.477694+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:25.211711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.123544+0000","last_change":"2024-03-17T23:54:56.354451+0000","last_active":"2024-03-17T23:56:54.123544+0000","last_peered":"2024-03-17T23:56:54.123544+0000","last_clean":"2024-03-17T23:56:54.123544+0000","last_became_active":"2024-03-17T23:54:56.354160+0000","last_became_peered":"2024-03-17T23:54:56.354160+0000","last_unstale":"2024-03-17T23:56:54.123544+0000","last_undegraded":"2024-03-17T23:56:54.123544+0000","last_fullsized":"2024-03-17T23:56:54.123544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:07:37.644444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.126588+0000","last_change":"2024-03-17T23:54:56.353705+0000","last_active":"2024-03-17T23:56:54.126588+0000","last_peered":"2024-03-17T23:56:54.126588+0000","last_clean":"2024-03-17T23:56:54.126588+0000","last_became_active":"2024-03-17T23:54:56.352389+0000","last_became_peered":"2024-03-17T23:54:56.352389+0000","last_unstale":"2024-03-17T23:56:54.126588+0000","last_undegraded":"2024-03-17T23:56:54.126588+0000","last_fullsized":"2024-03-17T23:56:54.126588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:17.153753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.115867+0000","last_change":"2024-03-17T23:54:56.348467+0000","last_active":"2024-03-17T23:56:54.115867+0000","last_peered":"2024-03-17T23:56:54.115867+0000","last_clean":"2024-03-17T23:56:54.115867+0000","last_became_active":"2024-03-17T23:54:56.348118+0000","last_became_peered":"2024-03-17T23:54:56.348118+0000","last_unstale":"2024-03-17T23:56:54.115867+0000","last_undegraded":"2024-03-17T23:56:54.115867+0000","last_fullsized":"2024-03-17T23:56:54.115867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:53:45.717303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.467820+0000","last_change":"2024-03-17T23:54:56.354117+0000","last_active":"2024-03-17T23:55:55.467820+0000","last_peered":"2024-03-17T23:55:55.467820+0000","last_clean":"2024-03-17T23:55:55.467820+0000","last_became_active":"2024-03-17T23:54:56.353818+0000","last_became_peered":"2024-03-17T23:54:56.353818+0000","last_unstale":"2024-03-17T23:55:55.467820+0000","last_undegraded":"2024-03-17T23:55:55.467820+0000","last_fullsized":"2024-03-17T23:55:55.467820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:09:17.314024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.739417+0000","last_change":"2024-03-17T23:54:56.363781+0000","last_active":"2024-03-17T23:56:54.739417+0000","last_peered":"2024-03-17T23:56:54.739417+0000","last_clean":"2024-03-17T23:56:54.739417+0000","last_became_active":"2024-03-17T23:54:56.363364+0000","last_became_peered":"2024-03-17T23:54:56.363364+0000","last_unstale":"2024-03-17T23:56:54.739417+0000","last_undegraded":"2024-03-17T23:56:54.739417+0000","last_fullsized":"2024-03-17T23:56:54.739417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:21:00.395329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.467216+0000","last_change":"2024-03-17T23:54:56.360905+0000","last_active":"2024-03-17T23:55:55.467216+0000","last_peered":"2024-03-17T23:55:55.467216+0000","last_clean":"2024-03-17T23:55:55.467216+0000","last_became_active":"2024-03-17T23:54:56.360492+0000","last_became_peered":"2024-03-17T23:54:56.360492+0000","last_unstale":"2024-03-17T23:55:55.467216+0000","last_undegraded":"2024-03-17T23:55:55.467216+0000","last_fullsized":"2024-03-17T23:55:55.467216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:34:42.440635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466685+0000","last_change":"2024-03-17T23:55:01.889086+0000","last_active":"2024-03-17T23:55:55.466685+0000","last_peered":"2024-03-17T23:55:55.466685+0000","last_clean":"2024-03-17T23:55:55.466685+0000","last_became_active":"2024-03-17T23:55:01.888505+0000","last_became_peered":"2024-03-17T23:55:01.888505+0000","last_unstale":"2024-03-17T23:55:55.466685+0000","last_undegraded":"2024-03-17T23:55:55.466685+0000","last_fullsized":"2024-03-17T23:55:55.466685+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:00:43.490360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":132,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.740390+0000","last_change":"2024-03-17T23:54:55.858790+0000","last_active":"2024-03-17T23:56:54.740390+0000","last_peered":"2024-03-17T23:56:54.740390+0000","last_clean":"2024-03-17T23:56:54.740390+0000","last_became_active":"2024-03-17T23:54:55.858437+0000","last_became_peered":"2024-03-17T23:54:55.858437+0000","last_unstale":"2024-03-17T23:56:54.740390+0000","last_undegraded":"2024-03-17T23:56:54.740390+0000","last_fullsized":"2024-03-17T23:56:54.740390+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:49:26.830471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.482904+0000","last_change":"2024-03-17T23:54:56.356641+0000","last_active":"2024-03-17T23:55:55.482904+0000","last_peered":"2024-03-17T23:55:55.482904+0000","last_clean":"2024-03-17T23:55:55.482904+0000","last_became_active":"2024-03-17T23:54:56.355402+0000","last_became_peered":"2024-03-17T23:54:56.355402+0000","last_unstale":"2024-03-17T23:55:55.482904+0000","last_undegraded":"2024-03-17T23:55:55.482904+0000","last_fullsized":"2024-03-17T23:55:55.482904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:01:04.699357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470595+0000","last_change":"2024-03-17T23:55:01.888632+0000","last_active":"2024-03-17T23:55:55.470595+0000","last_peered":"2024-03-17T23:55:55.470595+0000","last_clean":"2024-03-17T23:55:55.470595+0000","last_became_active":"2024-03-17T23:55:01.888240+0000","last_became_peered":"2024-03-17T23:55:01.888240+0000","last_unstale":"2024-03-17T23:55:55.470595+0000","last_undegraded":"2024-03-17T23:55:55.470595+0000","last_fullsized":"2024-03-17T23:55:55.470595+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:02:04.562112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.483028+0000","last_change":"2024-03-17T23:54:55.871842+0000","last_active":"2024-03-17T23:55:55.483028+0000","last_peered":"2024-03-17T23:55:55.483028+0000","last_clean":"2024-03-17T23:55:55.483028+0000","last_became_active":"2024-03-17T23:54:55.870965+0000","last_became_peered":"2024-03-17T23:54:55.870965+0000","last_unstale":"2024-03-17T23:55:55.483028+0000","last_undegraded":"2024-03-17T23:55:55.483028+0000","last_fullsized":"2024-03-17T23:55:55.483028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:27:04.762902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.130333+0000","last_change":"2024-03-17T23:54:56.368277+0000","last_active":"2024-03-17T23:56:54.130333+0000","last_peered":"2024-03-17T23:56:54.130333+0000","last_clean":"2024-03-17T23:56:54.130333+0000","last_became_active":"2024-03-17T23:54:56.366229+0000","last_became_peered":"2024-03-17T23:54:56.366229+0000","last_unstale":"2024-03-17T23:56:54.130333+0000","last_undegraded":"2024-03-17T23:56:54.130333+0000","last_fullsized":"2024-03-17T23:56:54.130333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:50:05.241745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.469885+0000","last_change":"2024-03-17T23:55:01.881731+0000","last_active":"2024-03-17T23:55:55.469885+0000","last_peered":"2024-03-17T23:55:55.469885+0000","last_clean":"2024-03-17T23:55:55.469885+0000","last_became_active":"2024-03-17T23:55:01.881383+0000","last_became_peered":"2024-03-17T23:55:01.881383+0000","last_unstale":"2024-03-17T23:55:55.469885+0000","last_undegraded":"2024-03-17T23:55:55.469885+0000","last_fullsized":"2024-03-17T23:55:55.469885+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:45.095672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.465103+0000","last_change":"2024-03-17T23:54:55.838014+0000","last_active":"2024-03-17T23:55:55.465103+0000","last_peered":"2024-03-17T23:55:55.465103+0000","last_clean":"2024-03-17T23:55:55.465103+0000","last_became_active":"2024-03-17T23:54:55.837694+0000","last_became_peered":"2024-03-17T23:54:55.837694+0000","last_unstale":"2024-03-17T23:55:55.465103+0000","last_undegraded":"2024-03-17T23:55:55.465103+0000","last_fullsized":"2024-03-17T23:55:55.465103+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:16:01.173460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.115707+0000","last_change":"2024-03-17T23:54:56.343240+0000","last_active":"2024-03-17T23:56:54.115707+0000","last_peered":"2024-03-17T23:56:54.115707+0000","last_clean":"2024-03-17T23:56:54.115707+0000","last_became_active":"2024-03-17T23:54:56.342928+0000","last_became_peered":"2024-03-17T23:54:56.342928+0000","last_unstale":"2024-03-17T23:56:54.115707+0000","last_undegraded":"2024-03-17T23:56:54.115707+0000","last_fullsized":"2024-03-17T23:56:54.115707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:33:43.922886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.469658+0000","last_change":"2024-03-17T23:55:01.880207+0000","last_active":"2024-03-17T23:55:55.469658+0000","last_peered":"2024-03-17T23:55:55.469658+0000","last_clean":"2024-03-17T23:55:55.469658+0000","last_became_active":"2024-03-17T23:55:01.879748+0000","last_became_peered":"2024-03-17T23:55:01.879748+0000","last_unstale":"2024-03-17T23:55:55.469658+0000","last_undegraded":"2024-03-17T23:55:55.469658+0000","last_fullsized":"2024-03-17T23:55:55.469658+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:33.782782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.464916+0000","last_change":"2024-03-17T23:54:55.839822+0000","last_active":"2024-03-17T23:55:55.464916+0000","last_peered":"2024-03-17T23:55:55.464916+0000","last_clean":"2024-03-17T23:55:55.464916+0000","last_became_active":"2024-03-17T23:54:55.839506+0000","last_became_peered":"2024-03-17T23:54:55.839506+0000","last_unstale":"2024-03-17T23:55:55.464916+0000","last_undegraded":"2024-03-17T23:55:55.464916+0000","last_fullsized":"2024-03-17T23:55:55.464916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:08.743582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470977+0000","last_change":"2024-03-17T23:54:56.346914+0000","last_active":"2024-03-17T23:55:55.470977+0000","last_peered":"2024-03-17T23:55:55.470977+0000","last_clean":"2024-03-17T23:55:55.470977+0000","last_became_active":"2024-03-17T23:54:56.346620+0000","last_became_peered":"2024-03-17T23:54:56.346620+0000","last_unstale":"2024-03-17T23:55:55.470977+0000","last_undegraded":"2024-03-17T23:55:55.470977+0000","last_fullsized":"2024-03-17T23:55:55.470977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:44:58.643614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.467093+0000","last_change":"2024-03-17T23:55:01.864527+0000","last_active":"2024-03-17T23:55:55.467093+0000","last_peered":"2024-03-17T23:55:55.467093+0000","last_clean":"2024-03-17T23:55:55.467093+0000","last_became_active":"2024-03-17T23:55:01.864106+0000","last_became_peered":"2024-03-17T23:55:01.864106+0000","last_unstale":"2024-03-17T23:55:55.467093+0000","last_undegraded":"2024-03-17T23:55:55.467093+0000","last_fullsized":"2024-03-17T23:55:55.467093+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:27:19.811639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":121,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:51.127853+0000","last_change":"2024-03-17T23:54:55.842156+0000","last_active":"2024-03-17T23:56:51.127853+0000","last_peered":"2024-03-17T23:56:51.127853+0000","last_clean":"2024-03-17T23:56:51.127853+0000","last_became_active":"2024-03-17T23:54:55.841858+0000","last_became_peered":"2024-03-17T23:54:55.841858+0000","last_unstale":"2024-03-17T23:56:51.127853+0000","last_undegraded":"2024-03-17T23:56:51.127853+0000","last_fullsized":"2024-03-17T23:56:51.127853+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:01:05.967638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.100617+0000","last_change":"2024-03-17T23:54:56.357770+0000","last_active":"2024-03-17T23:56:54.100617+0000","last_peered":"2024-03-17T23:56:54.100617+0000","last_clean":"2024-03-17T23:56:54.100617+0000","last_became_active":"2024-03-17T23:54:56.357411+0000","last_became_peered":"2024-03-17T23:54:56.357411+0000","last_unstale":"2024-03-17T23:56:54.100617+0000","last_undegraded":"2024-03-17T23:56:54.100617+0000","last_fullsized":"2024-03-17T23:56:54.100617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:01:04.685138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.467219+0000","last_change":"2024-03-17T23:55:01.883104+0000","last_active":"2024-03-17T23:55:55.467219+0000","last_peered":"2024-03-17T23:55:55.467219+0000","last_clean":"2024-03-17T23:55:55.467219+0000","last_became_active":"2024-03-17T23:55:01.882650+0000","last_became_peered":"2024-03-17T23:55:01.882650+0000","last_unstale":"2024-03-17T23:55:55.467219+0000","last_undegraded":"2024-03-17T23:55:55.467219+0000","last_fullsized":"2024-03-17T23:55:55.467219+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:54:18.420997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:55.132874+0000","last_change":"2024-03-17T23:55:55.576887+0000","last_active":"2024-03-17T23:56:55.132874+0000","last_peered":"2024-03-17T23:56:55.132874+0000","last_clean":"2024-03-17T23:56:55.132874+0000","last_became_active":"2024-03-17T23:55:55.572572+0000","last_became_peered":"2024-03-17T23:55:55.572572+0000","last_unstale":"2024-03-17T23:56:55.132874+0000","last_undegraded":"2024-03-17T23:56:55.132874+0000","last_fullsized":"2024-03-17T23:56:55.132874+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:47:44.693051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470933+0000","last_change":"2024-03-17T23:54:56.353311+0000","last_active":"2024-03-17T23:55:55.470933+0000","last_peered":"2024-03-17T23:55:55.470933+0000","last_clean":"2024-03-17T23:55:55.470933+0000","last_became_active":"2024-03-17T23:54:56.353013+0000","last_became_peered":"2024-03-17T23:54:56.353013+0000","last_unstale":"2024-03-17T23:55:55.470933+0000","last_undegraded":"2024-03-17T23:55:55.470933+0000","last_fullsized":"2024-03-17T23:55:55.470933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:21:20.119186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.497661+0000","last_change":"2024-03-17T23:55:55.497661+0000","last_active":"2024-03-17T23:55:55.497661+0000","last_peered":"2024-03-17T23:55:55.497661+0000","last_clean":"2024-03-17T23:55:55.497661+0000","last_became_active":"2024-03-17T23:55:55.497263+0000","last_became_peered":"2024-03-17T23:55:55.497263+0000","last_unstale":"2024-03-17T23:55:55.497661+0000","last_undegraded":"2024-03-17T23:55:55.497661+0000","last_fullsized":"2024-03-17T23:55:55.497661+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:15:23.555869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.481012+0000","last_change":"2024-03-17T23:54:55.862756+0000","last_active":"2024-03-17T23:55:55.481012+0000","last_peered":"2024-03-17T23:55:55.481012+0000","last_clean":"2024-03-17T23:55:55.481012+0000","last_became_active":"2024-03-17T23:54:55.862298+0000","last_became_peered":"2024-03-17T23:54:55.862298+0000","last_unstale":"2024-03-17T23:55:55.481012+0000","last_undegraded":"2024-03-17T23:55:55.481012+0000","last_fullsized":"2024-03-17T23:55:55.481012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:15:52.613136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.480429+0000","last_change":"2024-03-17T23:54:56.354359+0000","last_active":"2024-03-17T23:55:55.480429+0000","last_peered":"2024-03-17T23:55:55.480429+0000","last_clean":"2024-03-17T23:55:55.480429+0000","last_became_active":"2024-03-17T23:54:56.354002+0000","last_became_peered":"2024-03-17T23:54:56.354002+0000","last_unstale":"2024-03-17T23:55:55.480429+0000","last_undegraded":"2024-03-17T23:55:55.480429+0000","last_fullsized":"2024-03-17T23:55:55.480429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:41:12.128471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.493005+0000","last_change":"2024-03-17T23:55:01.880579+0000","last_active":"2024-03-17T23:55:55.493005+0000","last_peered":"2024-03-17T23:55:55.493005+0000","last_clean":"2024-03-17T23:55:55.493005+0000","last_became_active":"2024-03-17T23:55:01.879471+0000","last_became_peered":"2024-03-17T23:55:01.879471+0000","last_unstale":"2024-03-17T23:55:55.493005+0000","last_undegraded":"2024-03-17T23:55:55.493005+0000","last_fullsized":"2024-03-17T23:55:55.493005+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:51.117904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470548+0000","last_change":"2024-03-17T23:54:55.842998+0000","last_active":"2024-03-17T23:55:55.470548+0000","last_peered":"2024-03-17T23:55:55.470548+0000","last_clean":"2024-03-17T23:55:55.470548+0000","last_became_active":"2024-03-17T23:54:55.842678+0000","last_became_peered":"2024-03-17T23:54:55.842678+0000","last_unstale":"2024-03-17T23:55:55.470548+0000","last_undegraded":"2024-03-17T23:55:55.470548+0000","last_fullsized":"2024-03-17T23:55:55.470548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:14.546738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.738195+0000","last_change":"2024-03-17T23:54:56.365881+0000","last_active":"2024-03-17T23:56:54.738195+0000","last_peered":"2024-03-17T23:56:54.738195+0000","last_clean":"2024-03-17T23:56:54.738195+0000","last_became_active":"2024-03-17T23:54:56.365560+0000","last_became_peered":"2024-03-17T23:54:56.365560+0000","last_unstale":"2024-03-17T23:56:54.738195+0000","last_undegraded":"2024-03-17T23:56:54.738195+0000","last_fullsized":"2024-03-17T23:56:54.738195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:38.436644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.481311+0000","last_change":"2024-03-17T23:55:01.882651+0000","last_active":"2024-03-17T23:55:55.481311+0000","last_peered":"2024-03-17T23:55:55.481311+0000","last_clean":"2024-03-17T23:55:55.481311+0000","last_became_active":"2024-03-17T23:55:01.881717+0000","last_became_peered":"2024-03-17T23:55:01.881717+0000","last_unstale":"2024-03-17T23:55:55.481311+0000","last_undegraded":"2024-03-17T23:55:55.481311+0000","last_fullsized":"2024-03-17T23:55:55.481311+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:21:08.644184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'5","reported_seq":212,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:55.133523+0000","last_change":"2024-03-17T23:54:55.854822+0000","last_active":"2024-03-17T23:56:55.133523+0000","last_peered":"2024-03-17T23:56:55.133523+0000","last_clean":"2024-03-17T23:56:55.133523+0000","last_became_active":"2024-03-17T23:54:55.854501+0000","last_became_peered":"2024-03-17T23:54:55.854501+0000","last_unstale":"2024-03-17T23:56:55.133523+0000","last_undegraded":"2024-03-17T23:56:55.133523+0000","last_fullsized":"2024-03-17T23:56:55.133523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:04:12.504798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":176,"num_read_kb":176,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.109077+0000","last_change":"2024-03-17T23:54:56.361898+0000","last_active":"2024-03-17T23:56:54.109077+0000","last_peered":"2024-03-17T23:56:54.109077+0000","last_clean":"2024-03-17T23:56:54.109077+0000","last_became_active":"2024-03-17T23:54:56.361556+0000","last_became_peered":"2024-03-17T23:54:56.361556+0000","last_unstale":"2024-03-17T23:56:54.109077+0000","last_undegraded":"2024-03-17T23:56:54.109077+0000","last_fullsized":"2024-03-17T23:56:54.109077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:52:08.932801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.739027+0000","last_change":"2024-03-17T23:55:01.882628+0000","last_active":"2024-03-17T23:56:54.739027+0000","last_peered":"2024-03-17T23:56:54.739027+0000","last_clean":"2024-03-17T23:56:54.739027+0000","last_became_active":"2024-03-17T23:55:01.881886+0000","last_became_peered":"2024-03-17T23:55:01.881886+0000","last_unstale":"2024-03-17T23:56:54.739027+0000","last_undegraded":"2024-03-17T23:56:54.739027+0000","last_fullsized":"2024-03-17T23:56:54.739027+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:22:58.534442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571317+0000","last_change":"2024-03-17T23:54:55.839925+0000","last_active":"2024-03-17T23:55:55.571317+0000","last_peered":"2024-03-17T23:55:55.571317+0000","last_clean":"2024-03-17T23:55:55.571317+0000","last_became_active":"2024-03-17T23:54:55.839626+0000","last_became_peered":"2024-03-17T23:54:55.839626+0000","last_unstale":"2024-03-17T23:55:55.571317+0000","last_undegraded":"2024-03-17T23:55:55.571317+0000","last_fullsized":"2024-03-17T23:55:55.571317+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:39.456462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.123795+0000","last_change":"2024-03-17T23:54:56.352766+0000","last_active":"2024-03-17T23:56:54.123795+0000","last_peered":"2024-03-17T23:56:54.123795+0000","last_clean":"2024-03-17T23:56:54.123795+0000","last_became_active":"2024-03-17T23:54:56.351817+0000","last_became_peered":"2024-03-17T23:54:56.351817+0000","last_unstale":"2024-03-17T23:56:54.123795+0000","last_undegraded":"2024-03-17T23:56:54.123795+0000","last_fullsized":"2024-03-17T23:56:54.123795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:57:38.097163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.572538+0000","last_change":"2024-03-17T23:55:55.572538+0000","last_active":"2024-03-17T23:55:55.572538+0000","last_peered":"2024-03-17T23:55:55.572538+0000","last_clean":"2024-03-17T23:55:55.572538+0000","last_became_active":"2024-03-17T23:55:55.572138+0000","last_became_peered":"2024-03-17T23:55:55.572138+0000","last_unstale":"2024-03-17T23:55:55.572538+0000","last_undegraded":"2024-03-17T23:55:55.572538+0000","last_fullsized":"2024-03-17T23:55:55.572538+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:20:39.628674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470989+0000","last_change":"2024-03-17T23:54:55.863264+0000","last_active":"2024-03-17T23:55:55.470989+0000","last_peered":"2024-03-17T23:55:55.470989+0000","last_clean":"2024-03-17T23:55:55.470989+0000","last_became_active":"2024-03-17T23:54:55.863000+0000","last_became_peered":"2024-03-17T23:54:55.863000+0000","last_unstale":"2024-03-17T23:55:55.470989+0000","last_undegraded":"2024-03-17T23:55:55.470989+0000","last_fullsized":"2024-03-17T23:55:55.470989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:58.208064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.110276+0000","last_change":"2024-03-17T23:54:56.351161+0000","last_active":"2024-03-17T23:56:54.110276+0000","last_peered":"2024-03-17T23:56:54.110276+0000","last_clean":"2024-03-17T23:56:54.110276+0000","last_became_active":"2024-03-17T23:54:56.349787+0000","last_became_peered":"2024-03-17T23:54:56.349787+0000","last_unstale":"2024-03-17T23:56:54.110276+0000","last_undegraded":"2024-03-17T23:56:54.110276+0000","last_fullsized":"2024-03-17T23:56:54.110276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:19:28.635060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.738500+0000","last_change":"2024-03-17T23:55:55.500462+0000","last_active":"2024-03-17T23:56:54.738500+0000","last_peered":"2024-03-17T23:56:54.738500+0000","last_clean":"2024-03-17T23:56:54.738500+0000","last_became_active":"2024-03-17T23:55:55.499357+0000","last_became_peered":"2024-03-17T23:55:55.499357+0000","last_unstale":"2024-03-17T23:56:54.738500+0000","last_undegraded":"2024-03-17T23:56:54.738500+0000","last_fullsized":"2024-03-17T23:56:54.738500+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:18:28.701636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'38","reported_seq":254,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:55.133800+0000","last_change":"2024-03-17T23:54:55.846449+0000","last_active":"2024-03-17T23:56:55.133800+0000","last_peered":"2024-03-17T23:56:55.133800+0000","last_clean":"2024-03-17T23:56:55.133800+0000","last_became_active":"2024-03-17T23:54:55.846064+0000","last_became_peered":"2024-03-17T23:54:55.846064+0000","last_unstale":"2024-03-17T23:56:55.133800+0000","last_undegraded":"2024-03-17T23:56:55.133800+0000","last_fullsized":"2024-03-17T23:56:55.133800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:45.985724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64455,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":185,"num_read_kb":156,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.090254+0000","last_change":"2024-03-17T23:54:56.343717+0000","last_active":"2024-03-17T23:56:54.090254+0000","last_peered":"2024-03-17T23:56:54.090254+0000","last_clean":"2024-03-17T23:56:54.090254+0000","last_became_active":"2024-03-17T23:54:56.343418+0000","last_became_peered":"2024-03-17T23:54:56.343418+0000","last_unstale":"2024-03-17T23:56:54.090254+0000","last_undegraded":"2024-03-17T23:56:54.090254+0000","last_fullsized":"2024-03-17T23:56:54.090254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:11:40.402878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.471237+0000","last_change":"2024-03-17T23:55:01.884671+0000","last_active":"2024-03-17T23:55:55.471237+0000","last_peered":"2024-03-17T23:55:55.471237+0000","last_clean":"2024-03-17T23:55:55.471237+0000","last_became_active":"2024-03-17T23:55:01.884102+0000","last_became_peered":"2024-03-17T23:55:01.884102+0000","last_unstale":"2024-03-17T23:55:55.471237+0000","last_undegraded":"2024-03-17T23:55:55.471237+0000","last_fullsized":"2024-03-17T23:55:55.471237+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:53:31.198466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.504358+0000","last_change":"2024-03-17T23:55:55.503973+0000","last_active":"2024-03-17T23:55:55.504358+0000","last_peered":"2024-03-17T23:55:55.504358+0000","last_clean":"2024-03-17T23:55:55.504358+0000","last_became_active":"2024-03-17T23:55:55.488988+0000","last_became_peered":"2024-03-17T23:55:55.488988+0000","last_unstale":"2024-03-17T23:55:55.504358+0000","last_undegraded":"2024-03-17T23:55:55.504358+0000","last_fullsized":"2024-03-17T23:55:55.504358+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:42:13.924400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470807+0000","last_change":"2024-03-17T23:54:56.361699+0000","last_active":"2024-03-17T23:55:55.470807+0000","last_peered":"2024-03-17T23:55:55.470807+0000","last_clean":"2024-03-17T23:55:55.470807+0000","last_became_active":"2024-03-17T23:54:56.361365+0000","last_became_peered":"2024-03-17T23:54:56.361365+0000","last_unstale":"2024-03-17T23:55:55.470807+0000","last_undegraded":"2024-03-17T23:55:55.470807+0000","last_fullsized":"2024-03-17T23:55:55.470807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:24:30.649956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466789+0000","last_change":"2024-03-17T23:55:01.888676+0000","last_active":"2024-03-17T23:55:55.466789+0000","last_peered":"2024-03-17T23:55:55.466789+0000","last_clean":"2024-03-17T23:55:55.466789+0000","last_became_active":"2024-03-17T23:55:01.888224+0000","last_became_peered":"2024-03-17T23:55:01.888224+0000","last_unstale":"2024-03-17T23:55:55.466789+0000","last_undegraded":"2024-03-17T23:55:55.466789+0000","last_fullsized":"2024-03-17T23:55:55.466789+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:22.075096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470689+0000","last_change":"2024-03-17T23:54:55.857947+0000","last_active":"2024-03-17T23:55:55.470689+0000","last_peered":"2024-03-17T23:55:55.470689+0000","last_clean":"2024-03-17T23:55:55.470689+0000","last_became_active":"2024-03-17T23:54:55.857595+0000","last_became_peered":"2024-03-17T23:54:55.857595+0000","last_unstale":"2024-03-17T23:55:55.470689+0000","last_undegraded":"2024-03-17T23:55:55.470689+0000","last_fullsized":"2024-03-17T23:55:55.470689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:25.433053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571216+0000","last_change":"2024-03-17T23:54:56.347220+0000","last_active":"2024-03-17T23:55:55.571216+0000","last_peered":"2024-03-17T23:55:55.571216+0000","last_clean":"2024-03-17T23:55:55.571216+0000","last_became_active":"2024-03-17T23:54:56.346600+0000","last_became_peered":"2024-03-17T23:54:56.346600+0000","last_unstale":"2024-03-17T23:55:55.571216+0000","last_undegraded":"2024-03-17T23:55:55.571216+0000","last_fullsized":"2024-03-17T23:55:55.571216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:38:25.575448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.740108+0000","last_change":"2024-03-17T23:55:01.887016+0000","last_active":"2024-03-17T23:56:54.740108+0000","last_peered":"2024-03-17T23:56:54.740108+0000","last_clean":"2024-03-17T23:56:54.740108+0000","last_became_active":"2024-03-17T23:55:01.886576+0000","last_became_peered":"2024-03-17T23:55:01.886576+0000","last_unstale":"2024-03-17T23:56:54.740108+0000","last_undegraded":"2024-03-17T23:56:54.740108+0000","last_fullsized":"2024-03-17T23:56:54.740108+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:41:26.995982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":123,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:53.206485+0000","last_change":"2024-03-17T23:54:55.840254+0000","last_active":"2024-03-17T23:56:53.206485+0000","last_peered":"2024-03-17T23:56:53.206485+0000","last_clean":"2024-03-17T23:56:53.206485+0000","last_became_active":"2024-03-17T23:54:55.839740+0000","last_became_peered":"2024-03-17T23:54:55.839740+0000","last_unstale":"2024-03-17T23:56:53.206485+0000","last_undegraded":"2024-03-17T23:56:53.206485+0000","last_fullsized":"2024-03-17T23:56:53.206485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:39.664083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":90,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.570499+0000","last_change":"2024-03-17T23:54:56.343262+0000","last_active":"2024-03-17T23:55:55.570499+0000","last_peered":"2024-03-17T23:55:55.570499+0000","last_clean":"2024-03-17T23:55:55.570499+0000","last_became_active":"2024-03-17T23:54:56.342924+0000","last_became_peered":"2024-03-17T23:54:56.342924+0000","last_unstale":"2024-03-17T23:55:55.570499+0000","last_undegraded":"2024-03-17T23:55:55.570499+0000","last_fullsized":"2024-03-17T23:55:55.570499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:35:19.074001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.476494+0000","last_change":"2024-03-17T23:55:01.885081+0000","last_active":"2024-03-17T23:55:55.476494+0000","last_peered":"2024-03-17T23:55:55.476494+0000","last_clean":"2024-03-17T23:55:55.476494+0000","last_became_active":"2024-03-17T23:55:01.883328+0000","last_became_peered":"2024-03-17T23:55:01.883328+0000","last_unstale":"2024-03-17T23:55:55.476494+0000","last_undegraded":"2024-03-17T23:55:55.476494+0000","last_fullsized":"2024-03-17T23:55:55.476494+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:12:10.206774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.510896+0000","last_change":"2024-03-17T23:55:55.510896+0000","last_active":"2024-03-17T23:55:55.510896+0000","last_peered":"2024-03-17T23:55:55.510896+0000","last_clean":"2024-03-17T23:55:55.510896+0000","last_became_active":"2024-03-17T23:55:55.494373+0000","last_became_peered":"2024-03-17T23:55:55.494373+0000","last_unstale":"2024-03-17T23:55:55.510896+0000","last_undegraded":"2024-03-17T23:55:55.510896+0000","last_fullsized":"2024-03-17T23:55:55.510896+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:28:33.215554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470605+0000","last_change":"2024-03-17T23:54:56.345946+0000","last_active":"2024-03-17T23:55:55.470605+0000","last_peered":"2024-03-17T23:55:55.470605+0000","last_clean":"2024-03-17T23:55:55.470605+0000","last_became_active":"2024-03-17T23:54:56.345620+0000","last_became_peered":"2024-03-17T23:54:56.345620+0000","last_unstale":"2024-03-17T23:55:55.470605+0000","last_undegraded":"2024-03-17T23:55:55.470605+0000","last_fullsized":"2024-03-17T23:55:55.470605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:03:51.327966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470512+0000","last_change":"2024-03-17T23:55:01.884819+0000","last_active":"2024-03-17T23:55:55.470512+0000","last_peered":"2024-03-17T23:55:55.470512+0000","last_clean":"2024-03-17T23:55:55.470512+0000","last_became_active":"2024-03-17T23:55:01.884045+0000","last_became_peered":"2024-03-17T23:55:01.884045+0000","last_unstale":"2024-03-17T23:55:55.470512+0000","last_undegraded":"2024-03-17T23:55:55.470512+0000","last_fullsized":"2024-03-17T23:55:55.470512+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:10.147132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.465237+0000","last_change":"2024-03-17T23:54:55.840370+0000","last_active":"2024-03-17T23:55:55.465237+0000","last_peered":"2024-03-17T23:55:55.465237+0000","last_clean":"2024-03-17T23:55:55.465237+0000","last_became_active":"2024-03-17T23:54:55.840049+0000","last_became_peered":"2024-03-17T23:54:55.840049+0000","last_unstale":"2024-03-17T23:55:55.465237+0000","last_undegraded":"2024-03-17T23:55:55.465237+0000","last_fullsized":"2024-03-17T23:55:55.465237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:40:46.523632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466733+0000","last_change":"2024-03-17T23:54:56.346533+0000","last_active":"2024-03-17T23:55:55.466733+0000","last_peered":"2024-03-17T23:55:55.466733+0000","last_clean":"2024-03-17T23:55:55.466733+0000","last_became_active":"2024-03-17T23:54:56.346214+0000","last_became_peered":"2024-03-17T23:54:56.346214+0000","last_unstale":"2024-03-17T23:55:55.466733+0000","last_undegraded":"2024-03-17T23:55:55.466733+0000","last_fullsized":"2024-03-17T23:55:55.466733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:06:59.017416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.465258+0000","last_change":"2024-03-17T23:55:02.318944+0000","last_active":"2024-03-17T23:55:55.465258+0000","last_peered":"2024-03-17T23:55:55.465258+0000","last_clean":"2024-03-17T23:55:55.465258+0000","last_became_active":"2024-03-17T23:55:02.318432+0000","last_became_peered":"2024-03-17T23:55:02.318432+0000","last_unstale":"2024-03-17T23:55:55.465258+0000","last_undegraded":"2024-03-17T23:55:55.465258+0000","last_fullsized":"2024-03-17T23:55:55.465258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:23:47.986701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.739030+0000","last_change":"2024-03-17T23:54:55.873746+0000","last_active":"2024-03-17T23:56:54.739030+0000","last_peered":"2024-03-17T23:56:54.739030+0000","last_clean":"2024-03-17T23:56:54.739030+0000","last_became_active":"2024-03-17T23:54:55.872229+0000","last_became_peered":"2024-03-17T23:54:55.872229+0000","last_unstale":"2024-03-17T23:56:54.739030+0000","last_undegraded":"2024-03-17T23:56:54.739030+0000","last_fullsized":"2024-03-17T23:56:54.739030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:10.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:55.133113+0000","last_change":"2024-03-17T23:54:56.361294+0000","last_active":"2024-03-17T23:56:55.133113+0000","last_peered":"2024-03-17T23:56:55.133113+0000","last_clean":"2024-03-17T23:56:55.133113+0000","last_became_active":"2024-03-17T23:54:56.360976+0000","last_became_peered":"2024-03-17T23:54:56.360976+0000","last_unstale":"2024-03-17T23:56:55.133113+0000","last_undegraded":"2024-03-17T23:56:55.133113+0000","last_fullsized":"2024-03-17T23:56:55.133113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:28:58.685991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.475376+0000","last_change":"2024-03-17T23:55:01.880221+0000","last_active":"2024-03-17T23:55:55.475376+0000","last_peered":"2024-03-17T23:55:55.475376+0000","last_clean":"2024-03-17T23:55:55.475376+0000","last_became_active":"2024-03-17T23:55:01.879785+0000","last_became_peered":"2024-03-17T23:55:01.879785+0000","last_unstale":"2024-03-17T23:55:55.475376+0000","last_undegraded":"2024-03-17T23:55:55.475376+0000","last_fullsized":"2024-03-17T23:55:55.475376+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:24:51.698857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":121,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:52.172935+0000","last_change":"2024-03-17T23:54:55.854176+0000","last_active":"2024-03-17T23:56:52.172935+0000","last_peered":"2024-03-17T23:56:52.172935+0000","last_clean":"2024-03-17T23:56:52.172935+0000","last_became_active":"2024-03-17T23:54:55.853859+0000","last_became_peered":"2024-03-17T23:54:55.853859+0000","last_unstale":"2024-03-17T23:56:52.172935+0000","last_undegraded":"2024-03-17T23:56:52.172935+0000","last_fullsized":"2024-03-17T23:56:52.172935+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:43:43.319635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:55.132931+0000","last_change":"2024-03-17T23:55:55.505626+0000","last_active":"2024-03-17T23:56:55.132931+0000","last_peered":"2024-03-17T23:56:55.132931+0000","last_clean":"2024-03-17T23:56:55.132931+0000","last_became_active":"2024-03-17T23:55:55.505284+0000","last_became_peered":"2024-03-17T23:55:55.505284+0000","last_unstale":"2024-03-17T23:56:55.132931+0000","last_undegraded":"2024-03-17T23:56:55.132931+0000","last_fullsized":"2024-03-17T23:56:55.132931+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:10:48.182303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.475236+0000","last_change":"2024-03-17T23:55:02.320494+0000","last_active":"2024-03-17T23:55:55.475236+0000","last_peered":"2024-03-17T23:55:55.475236+0000","last_clean":"2024-03-17T23:55:55.475236+0000","last_became_active":"2024-03-17T23:55:02.319188+0000","last_became_peered":"2024-03-17T23:55:02.319188+0000","last_unstale":"2024-03-17T23:55:55.475236+0000","last_undegraded":"2024-03-17T23:55:55.475236+0000","last_fullsized":"2024-03-17T23:55:55.475236+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:30:06.348718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470704+0000","last_change":"2024-03-17T23:54:56.340798+0000","last_active":"2024-03-17T23:55:55.470704+0000","last_peered":"2024-03-17T23:55:55.470704+0000","last_clean":"2024-03-17T23:55:55.470704+0000","last_became_active":"2024-03-17T23:54:56.340386+0000","last_became_peered":"2024-03-17T23:54:56.340386+0000","last_unstale":"2024-03-17T23:55:55.470704+0000","last_undegraded":"2024-03-17T23:55:55.470704+0000","last_fullsized":"2024-03-17T23:55:55.470704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:57:58.546464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":119,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:51.201916+0000","last_change":"2024-03-17T23:54:55.870475+0000","last_active":"2024-03-17T23:56:51.201916+0000","last_peered":"2024-03-17T23:56:51.201916+0000","last_clean":"2024-03-17T23:56:51.201916+0000","last_became_active":"2024-03-17T23:54:55.870119+0000","last_became_peered":"2024-03-17T23:54:55.870119+0000","last_unstale":"2024-03-17T23:56:51.201916+0000","last_undegraded":"2024-03-17T23:56:51.201916+0000","last_fullsized":"2024-03-17T23:56:51.201916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:12:15.421117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571442+0000","last_change":"2024-03-17T23:55:01.892759+0000","last_active":"2024-03-17T23:55:55.571442+0000","last_peered":"2024-03-17T23:55:55.571442+0000","last_clean":"2024-03-17T23:55:55.571442+0000","last_became_active":"2024-03-17T23:55:01.892240+0000","last_became_peered":"2024-03-17T23:55:01.892240+0000","last_unstale":"2024-03-17T23:55:55.571442+0000","last_undegraded":"2024-03-17T23:55:55.571442+0000","last_fullsized":"2024-03-17T23:55:55.571442+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:16:24.391971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.481738+0000","last_change":"2024-03-17T23:54:56.354898+0000","last_active":"2024-03-17T23:55:55.481738+0000","last_peered":"2024-03-17T23:55:55.481738+0000","last_clean":"2024-03-17T23:55:55.481738+0000","last_became_active":"2024-03-17T23:54:56.354345+0000","last_became_peered":"2024-03-17T23:54:56.354345+0000","last_unstale":"2024-03-17T23:55:55.481738+0000","last_undegraded":"2024-03-17T23:55:55.481738+0000","last_fullsized":"2024-03-17T23:55:55.481738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:18:56.962412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.469923+0000","last_change":"2024-03-17T23:54:55.846706+0000","last_active":"2024-03-17T23:55:55.469923+0000","last_peered":"2024-03-17T23:55:55.469923+0000","last_clean":"2024-03-17T23:55:55.469923+0000","last_became_active":"2024-03-17T23:54:55.846372+0000","last_became_peered":"2024-03-17T23:54:55.846372+0000","last_unstale":"2024-03-17T23:55:55.469923+0000","last_undegraded":"2024-03-17T23:55:55.469923+0000","last_fullsized":"2024-03-17T23:55:55.469923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:28:47.697975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":351,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.570430+0000","last_change":"2024-03-17T23:53:34.383413+0000","last_active":"2024-03-17T23:55:55.570430+0000","last_peered":"2024-03-17T23:55:55.570430+0000","last_clean":"2024-03-17T23:55:55.570430+0000","last_became_active":"2024-03-17T23:53:34.382875+0000","last_became_peered":"2024-03-17T23:53:34.382875+0000","last_unstale":"2024-03-17T23:55:55.570430+0000","last_undegraded":"2024-03-17T23:55:55.570430+0000","last_fullsized":"2024-03-17T23:55:55.570430+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:38.641018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:38.641018+0000","last_clean_scrub_stamp":"2024-03-17T23:49:38.641018+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:49:45.234628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.490220+0000","last_change":"2024-03-17T23:55:55.490220+0000","last_active":"2024-03-17T23:55:55.490220+0000","last_peered":"2024-03-17T23:55:55.490220+0000","last_clean":"2024-03-17T23:55:55.490220+0000","last_became_active":"2024-03-17T23:55:55.489804+0000","last_became_peered":"2024-03-17T23:55:55.489804+0000","last_unstale":"2024-03-17T23:55:55.490220+0000","last_undegraded":"2024-03-17T23:55:55.490220+0000","last_fullsized":"2024-03-17T23:55:55.490220+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:48.606538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.500027+0000","last_change":"2024-03-17T23:55:55.500027+0000","last_active":"2024-03-17T23:55:55.500027+0000","last_peered":"2024-03-17T23:55:55.500027+0000","last_clean":"2024-03-17T23:55:55.500027+0000","last_became_active":"2024-03-17T23:55:55.499624+0000","last_became_peered":"2024-03-17T23:55:55.499624+0000","last_unstale":"2024-03-17T23:55:55.500027+0000","last_undegraded":"2024-03-17T23:55:55.500027+0000","last_fullsized":"2024-03-17T23:55:55.500027+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:15:07.830593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.570448+0000","last_change":"2024-03-17T23:54:55.840743+0000","last_active":"2024-03-17T23:55:55.570448+0000","last_peered":"2024-03-17T23:55:55.570448+0000","last_clean":"2024-03-17T23:55:55.570448+0000","last_became_active":"2024-03-17T23:54:55.840463+0000","last_became_peered":"2024-03-17T23:54:55.840463+0000","last_unstale":"2024-03-17T23:55:55.570448+0000","last_undegraded":"2024-03-17T23:55:55.570448+0000","last_fullsized":"2024-03-17T23:55:55.570448+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:24:24.130911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466913+0000","last_change":"2024-03-17T23:55:01.876895+0000","last_active":"2024-03-17T23:55:55.466913+0000","last_peered":"2024-03-17T23:55:55.466913+0000","last_clean":"2024-03-17T23:55:55.466913+0000","last_became_active":"2024-03-17T23:55:01.876443+0000","last_became_peered":"2024-03-17T23:55:01.876443+0000","last_unstale":"2024-03-17T23:55:55.466913+0000","last_undegraded":"2024-03-17T23:55:55.466913+0000","last_fullsized":"2024-03-17T23:55:55.466913+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:51:55.737167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.517893+0000","last_change":"2024-03-17T23:55:55.517534+0000","last_active":"2024-03-17T23:55:55.517893+0000","last_peered":"2024-03-17T23:55:55.517893+0000","last_clean":"2024-03-17T23:55:55.517893+0000","last_became_active":"2024-03-17T23:55:55.499568+0000","last_became_peered":"2024-03-17T23:55:55.499568+0000","last_unstale":"2024-03-17T23:55:55.517893+0000","last_undegraded":"2024-03-17T23:55:55.517893+0000","last_fullsized":"2024-03-17T23:55:55.517893+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:36:23.064322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.475674+0000","last_change":"2024-03-17T23:54:56.365913+0000","last_active":"2024-03-17T23:55:55.475674+0000","last_peered":"2024-03-17T23:55:55.475674+0000","last_clean":"2024-03-17T23:55:55.475674+0000","last_became_active":"2024-03-17T23:54:56.365578+0000","last_became_peered":"2024-03-17T23:54:56.365578+0000","last_unstale":"2024-03-17T23:55:55.475674+0000","last_undegraded":"2024-03-17T23:55:55.475674+0000","last_fullsized":"2024-03-17T23:55:55.475674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:49.330026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.480298+0000","last_change":"2024-03-17T23:55:02.319060+0000","last_active":"2024-03-17T23:55:55.480298+0000","last_peered":"2024-03-17T23:55:55.480298+0000","last_clean":"2024-03-17T23:55:55.480298+0000","last_became_active":"2024-03-17T23:55:02.318542+0000","last_became_peered":"2024-03-17T23:55:02.318542+0000","last_unstale":"2024-03-17T23:55:55.480298+0000","last_undegraded":"2024-03-17T23:55:55.480298+0000","last_fullsized":"2024-03-17T23:55:55.480298+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:59:27.334039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'14","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.739683+0000","last_change":"2024-03-17T23:54:55.861011+0000","last_active":"2024-03-17T23:56:54.739683+0000","last_peered":"2024-03-17T23:56:54.739683+0000","last_clean":"2024-03-17T23:56:54.739683+0000","last_became_active":"2024-03-17T23:54:55.860388+0000","last_became_peered":"2024-03-17T23:54:55.860388+0000","last_unstale":"2024-03-17T23:56:54.739683+0000","last_undegraded":"2024-03-17T23:56:54.739683+0000","last_fullsized":"2024-03-17T23:56:54.739683+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:20:11.943034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.483269+0000","last_change":"2024-03-17T23:54:56.357572+0000","last_active":"2024-03-17T23:55:55.483269+0000","last_peered":"2024-03-17T23:55:55.483269+0000","last_clean":"2024-03-17T23:55:55.483269+0000","last_became_active":"2024-03-17T23:54:56.357226+0000","last_became_peered":"2024-03-17T23:54:56.357226+0000","last_unstale":"2024-03-17T23:55:55.483269+0000","last_undegraded":"2024-03-17T23:55:55.483269+0000","last_fullsized":"2024-03-17T23:55:55.483269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:35:44.939454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466533+0000","last_change":"2024-03-17T23:55:01.860239+0000","last_active":"2024-03-17T23:55:55.466533+0000","last_peered":"2024-03-17T23:55:55.466533+0000","last_clean":"2024-03-17T23:55:55.466533+0000","last_became_active":"2024-03-17T23:55:01.859754+0000","last_became_peered":"2024-03-17T23:55:01.859754+0000","last_unstale":"2024-03-17T23:55:55.466533+0000","last_undegraded":"2024-03-17T23:55:55.466533+0000","last_fullsized":"2024-03-17T23:55:55.466533+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:30:16.833040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470570+0000","last_change":"2024-03-17T23:54:55.855917+0000","last_active":"2024-03-17T23:55:55.470570+0000","last_peered":"2024-03-17T23:55:55.470570+0000","last_clean":"2024-03-17T23:55:55.470570+0000","last_became_active":"2024-03-17T23:54:55.855562+0000","last_became_peered":"2024-03-17T23:54:55.855562+0000","last_unstale":"2024-03-17T23:55:55.470570+0000","last_undegraded":"2024-03-17T23:55:55.470570+0000","last_fullsized":"2024-03-17T23:55:55.470570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:44.928086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.130004+0000","last_change":"2024-03-17T23:54:56.362634+0000","last_active":"2024-03-17T23:56:54.130004+0000","last_peered":"2024-03-17T23:56:54.130004+0000","last_clean":"2024-03-17T23:56:54.130004+0000","last_became_active":"2024-03-17T23:54:56.361910+0000","last_became_peered":"2024-03-17T23:54:56.361910+0000","last_unstale":"2024-03-17T23:56:54.130004+0000","last_undegraded":"2024-03-17T23:56:54.130004+0000","last_fullsized":"2024-03-17T23:56:54.130004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:20:34.988074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.469836+0000","last_change":"2024-03-17T23:55:02.319090+0000","last_active":"2024-03-17T23:55:55.469836+0000","last_peered":"2024-03-17T23:55:55.469836+0000","last_clean":"2024-03-17T23:55:55.469836+0000","last_became_active":"2024-03-17T23:55:02.318695+0000","last_became_peered":"2024-03-17T23:55:02.318695+0000","last_unstale":"2024-03-17T23:55:55.469836+0000","last_undegraded":"2024-03-17T23:55:55.469836+0000","last_fullsized":"2024-03-17T23:55:55.469836+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:32:13.984797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.570814+0000","last_change":"2024-03-17T23:54:55.852999+0000","last_active":"2024-03-17T23:55:55.570814+0000","last_peered":"2024-03-17T23:55:55.570814+0000","last_clean":"2024-03-17T23:55:55.570814+0000","last_became_active":"2024-03-17T23:54:55.852653+0000","last_became_peered":"2024-03-17T23:54:55.852653+0000","last_unstale":"2024-03-17T23:55:55.570814+0000","last_undegraded":"2024-03-17T23:55:55.570814+0000","last_fullsized":"2024-03-17T23:55:55.570814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:04:10.561807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.066103+0000","last_change":"2024-03-17T23:54:56.345171+0000","last_active":"2024-03-17T23:56:54.066103+0000","last_peered":"2024-03-17T23:56:54.066103+0000","last_clean":"2024-03-17T23:56:54.066103+0000","last_became_active":"2024-03-17T23:54:56.344146+0000","last_became_peered":"2024-03-17T23:54:56.344146+0000","last_unstale":"2024-03-17T23:56:54.066103+0000","last_undegraded":"2024-03-17T23:56:54.066103+0000","last_fullsized":"2024-03-17T23:56:54.066103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:08.546622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.740272+0000","last_change":"2024-03-17T23:55:01.882655+0000","last_active":"2024-03-17T23:56:54.740272+0000","last_peered":"2024-03-17T23:56:54.740272+0000","last_clean":"2024-03-17T23:56:54.740272+0000","last_became_active":"2024-03-17T23:55:01.882147+0000","last_became_peered":"2024-03-17T23:55:01.882147+0000","last_unstale":"2024-03-17T23:56:54.740272+0000","last_undegraded":"2024-03-17T23:56:54.740272+0000","last_fullsized":"2024-03-17T23:56:54.740272+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:17:55.653103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.480958+0000","last_change":"2024-03-17T23:54:55.862499+0000","last_active":"2024-03-17T23:55:55.480958+0000","last_peered":"2024-03-17T23:55:55.480958+0000","last_clean":"2024-03-17T23:55:55.480958+0000","last_became_active":"2024-03-17T23:54:55.862003+0000","last_became_peered":"2024-03-17T23:54:55.862003+0000","last_unstale":"2024-03-17T23:55:55.480958+0000","last_undegraded":"2024-03-17T23:55:55.480958+0000","last_fullsized":"2024-03-17T23:55:55.480958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:37:19.046543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466673+0000","last_change":"2024-03-17T23:54:56.352154+0000","last_active":"2024-03-17T23:55:55.466673+0000","last_peered":"2024-03-17T23:55:55.466673+0000","last_clean":"2024-03-17T23:55:55.466673+0000","last_became_active":"2024-03-17T23:54:56.351818+0000","last_became_peered":"2024-03-17T23:54:56.351818+0000","last_unstale":"2024-03-17T23:55:55.466673+0000","last_undegraded":"2024-03-17T23:55:55.466673+0000","last_fullsized":"2024-03-17T23:55:55.466673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:44:46.538778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.469859+0000","last_change":"2024-03-17T23:55:01.876605+0000","last_active":"2024-03-17T23:55:55.469859+0000","last_peered":"2024-03-17T23:55:55.469859+0000","last_clean":"2024-03-17T23:55:55.469859+0000","last_became_active":"2024-03-17T23:55:01.876253+0000","last_became_peered":"2024-03-17T23:55:01.876253+0000","last_unstale":"2024-03-17T23:55:55.469859+0000","last_undegraded":"2024-03-17T23:55:55.469859+0000","last_fullsized":"2024-03-17T23:55:55.469859+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:27:43.271798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466918+0000","last_change":"2024-03-17T23:54:55.862209+0000","last_active":"2024-03-17T23:55:55.466918+0000","last_peered":"2024-03-17T23:55:55.466918+0000","last_clean":"2024-03-17T23:55:55.466918+0000","last_became_active":"2024-03-17T23:54:55.861829+0000","last_became_peered":"2024-03-17T23:54:55.861829+0000","last_unstale":"2024-03-17T23:55:55.466918+0000","last_undegraded":"2024-03-17T23:55:55.466918+0000","last_fullsized":"2024-03-17T23:55:55.466918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:13:58.756538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.479975+0000","last_change":"2024-03-17T23:54:56.366869+0000","last_active":"2024-03-17T23:55:55.479975+0000","last_peered":"2024-03-17T23:55:55.479975+0000","last_clean":"2024-03-17T23:55:55.479975+0000","last_became_active":"2024-03-17T23:54:56.365380+0000","last_became_peered":"2024-03-17T23:54:56.365380+0000","last_unstale":"2024-03-17T23:55:55.479975+0000","last_undegraded":"2024-03-17T23:55:55.479975+0000","last_fullsized":"2024-03-17T23:55:55.479975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:28:03.010849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.469920+0000","last_change":"2024-03-17T23:55:02.318820+0000","last_active":"2024-03-17T23:55:55.469920+0000","last_peered":"2024-03-17T23:55:55.469920+0000","last_clean":"2024-03-17T23:55:55.469920+0000","last_became_active":"2024-03-17T23:55:02.318308+0000","last_became_peered":"2024-03-17T23:55:02.318308+0000","last_unstale":"2024-03-17T23:55:55.469920+0000","last_undegraded":"2024-03-17T23:55:55.469920+0000","last_fullsized":"2024-03-17T23:55:55.469920+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:40:35.224105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.464575+0000","last_change":"2024-03-17T23:54:55.837980+0000","last_active":"2024-03-17T23:55:55.464575+0000","last_peered":"2024-03-17T23:55:55.464575+0000","last_clean":"2024-03-17T23:55:55.464575+0000","last_became_active":"2024-03-17T23:54:55.837614+0000","last_became_peered":"2024-03-17T23:54:55.837614+0000","last_unstale":"2024-03-17T23:55:55.464575+0000","last_undegraded":"2024-03-17T23:55:55.464575+0000","last_fullsized":"2024-03-17T23:55:55.464575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:05:07.228281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571194+0000","last_change":"2024-03-17T23:54:56.348534+0000","last_active":"2024-03-17T23:55:55.571194+0000","last_peered":"2024-03-17T23:55:55.571194+0000","last_clean":"2024-03-17T23:55:55.571194+0000","last_became_active":"2024-03-17T23:54:56.348138+0000","last_became_peered":"2024-03-17T23:54:56.348138+0000","last_unstale":"2024-03-17T23:55:55.571194+0000","last_undegraded":"2024-03-17T23:55:55.571194+0000","last_fullsized":"2024-03-17T23:55:55.571194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:47:49.481608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571122+0000","last_change":"2024-03-17T23:55:01.876287+0000","last_active":"2024-03-17T23:55:55.571122+0000","last_peered":"2024-03-17T23:55:55.571122+0000","last_clean":"2024-03-17T23:55:55.571122+0000","last_became_active":"2024-03-17T23:55:01.875861+0000","last_became_peered":"2024-03-17T23:55:01.875861+0000","last_unstale":"2024-03-17T23:55:55.571122+0000","last_undegraded":"2024-03-17T23:55:55.571122+0000","last_fullsized":"2024-03-17T23:55:55.571122+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:33:59.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":122,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:52.128715+0000","last_change":"2024-03-17T23:54:55.870989+0000","last_active":"2024-03-17T23:56:52.128715+0000","last_peered":"2024-03-17T23:56:52.128715+0000","last_clean":"2024-03-17T23:56:52.128715+0000","last_became_active":"2024-03-17T23:54:55.870633+0000","last_became_peered":"2024-03-17T23:54:55.870633+0000","last_unstale":"2024-03-17T23:56:52.128715+0000","last_undegraded":"2024-03-17T23:56:52.128715+0000","last_fullsized":"2024-03-17T23:56:52.128715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:55.310289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571567+0000","last_change":"2024-03-17T23:54:56.345874+0000","last_active":"2024-03-17T23:55:55.571567+0000","last_peered":"2024-03-17T23:55:55.571567+0000","last_clean":"2024-03-17T23:55:55.571567+0000","last_became_active":"2024-03-17T23:54:56.345547+0000","last_became_peered":"2024-03-17T23:54:56.345547+0000","last_unstale":"2024-03-17T23:55:55.571567+0000","last_undegraded":"2024-03-17T23:55:55.571567+0000","last_fullsized":"2024-03-17T23:55:55.571567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:52:31.724224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.484602+0000","last_change":"2024-03-17T23:55:01.892947+0000","last_active":"2024-03-17T23:55:55.484602+0000","last_peered":"2024-03-17T23:55:55.484602+0000","last_clean":"2024-03-17T23:55:55.484602+0000","last_became_active":"2024-03-17T23:55:01.892481+0000","last_became_peered":"2024-03-17T23:55:01.892481+0000","last_unstale":"2024-03-17T23:55:55.484602+0000","last_undegraded":"2024-03-17T23:55:55.484602+0000","last_fullsized":"2024-03-17T23:55:55.484602+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:07:14.901490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.471113+0000","last_change":"2024-03-17T23:54:55.857943+0000","last_active":"2024-03-17T23:55:55.471113+0000","last_peered":"2024-03-17T23:55:55.471113+0000","last_clean":"2024-03-17T23:55:55.471113+0000","last_became_active":"2024-03-17T23:54:55.857592+0000","last_became_peered":"2024-03-17T23:54:55.857592+0000","last_unstale":"2024-03-17T23:55:55.471113+0000","last_undegraded":"2024-03-17T23:55:55.471113+0000","last_fullsized":"2024-03-17T23:55:55.471113+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:50:03.643518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571123+0000","last_change":"2024-03-17T23:54:56.357496+0000","last_active":"2024-03-17T23:55:55.571123+0000","last_peered":"2024-03-17T23:55:55.571123+0000","last_clean":"2024-03-17T23:55:55.571123+0000","last_became_active":"2024-03-17T23:54:56.357242+0000","last_became_peered":"2024-03-17T23:54:56.357242+0000","last_unstale":"2024-03-17T23:55:55.571123+0000","last_undegraded":"2024-03-17T23:55:55.571123+0000","last_fullsized":"2024-03-17T23:55:55.571123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:52:49.941623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470176+0000","last_change":"2024-03-17T23:55:01.866288+0000","last_active":"2024-03-17T23:55:55.470176+0000","last_peered":"2024-03-17T23:55:55.470176+0000","last_clean":"2024-03-17T23:55:55.470176+0000","last_became_active":"2024-03-17T23:55:01.865875+0000","last_became_peered":"2024-03-17T23:55:01.865875+0000","last_unstale":"2024-03-17T23:55:55.470176+0000","last_undegraded":"2024-03-17T23:55:55.470176+0000","last_fullsized":"2024-03-17T23:55:55.470176+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:55:00.822874+0000","last_clean_scrub_stamp":"2024-03-17T23:55:00.822874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:45:43.198830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.476761+0000","last_change":"2024-03-17T23:54:55.871971+0000","last_active":"2024-03-17T23:55:55.476761+0000","last_peered":"2024-03-17T23:55:55.476761+0000","last_clean":"2024-03-17T23:55:55.476761+0000","last_became_active":"2024-03-17T23:54:55.871434+0000","last_became_peered":"2024-03-17T23:54:55.871434+0000","last_unstale":"2024-03-17T23:55:55.476761+0000","last_undegraded":"2024-03-17T23:55:55.476761+0000","last_fullsized":"2024-03-17T23:55:55.476761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:54.767185+0000","last_clean_scrub_stamp":"2024-03-17T23:54:54.767185+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:41:10.667628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.571377+0000","last_change":"2024-03-17T23:54:56.343595+0000","last_active":"2024-03-17T23:55:55.571377+0000","last_peered":"2024-03-17T23:55:55.571377+0000","last_clean":"2024-03-17T23:55:55.571377+0000","last_became_active":"2024-03-17T23:54:56.343270+0000","last_became_peered":"2024-03-17T23:54:56.343270+0000","last_unstale":"2024-03-17T23:55:55.571377+0000","last_undegraded":"2024-03-17T23:55:55.571377+0000","last_fullsized":"2024-03-17T23:55:55.571377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:22:05.904023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.467028+0000","last_change":"2024-03-17T23:54:56.346571+0000","last_active":"2024-03-17T23:55:55.467028+0000","last_peered":"2024-03-17T23:55:55.467028+0000","last_clean":"2024-03-17T23:55:55.467028+0000","last_became_active":"2024-03-17T23:54:56.346234+0000","last_became_peered":"2024-03-17T23:54:56.346234+0000","last_unstale":"2024-03-17T23:55:55.467028+0000","last_undegraded":"2024-03-17T23:55:55.467028+0000","last_fullsized":"2024-03-17T23:55:55.467028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:25.448506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466928+0000","last_change":"2024-03-17T23:54:56.342299+0000","last_active":"2024-03-17T23:55:55.466928+0000","last_peered":"2024-03-17T23:55:55.466928+0000","last_clean":"2024-03-17T23:55:55.466928+0000","last_became_active":"2024-03-17T23:54:56.341988+0000","last_became_peered":"2024-03-17T23:54:56.341988+0000","last_unstale":"2024-03-17T23:55:55.466928+0000","last_undegraded":"2024-03-17T23:55:55.466928+0000","last_fullsized":"2024-03-17T23:55:55.466928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:03:38.595595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466805+0000","last_change":"2024-03-17T23:54:56.360601+0000","last_active":"2024-03-17T23:55:55.466805+0000","last_peered":"2024-03-17T23:55:55.466805+0000","last_clean":"2024-03-17T23:55:55.466805+0000","last_became_active":"2024-03-17T23:54:56.360215+0000","last_became_peered":"2024-03-17T23:54:56.360215+0000","last_unstale":"2024-03-17T23:55:55.466805+0000","last_undegraded":"2024-03-17T23:55:55.466805+0000","last_fullsized":"2024-03-17T23:55:55.466805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:42:07.661460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.131675+0000","last_change":"2024-03-17T23:54:56.358523+0000","last_active":"2024-03-17T23:56:54.131675+0000","last_peered":"2024-03-17T23:56:54.131675+0000","last_clean":"2024-03-17T23:56:54.131675+0000","last_became_active":"2024-03-17T23:54:56.358172+0000","last_became_peered":"2024-03-17T23:54:56.358172+0000","last_unstale":"2024-03-17T23:56:54.131675+0000","last_undegraded":"2024-03-17T23:56:54.131675+0000","last_fullsized":"2024-03-17T23:56:54.131675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:00.542176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.475571+0000","last_change":"2024-03-17T23:54:56.348619+0000","last_active":"2024-03-17T23:55:55.475571+0000","last_peered":"2024-03-17T23:55:55.475571+0000","last_clean":"2024-03-17T23:55:55.475571+0000","last_became_active":"2024-03-17T23:54:56.348309+0000","last_became_peered":"2024-03-17T23:54:56.348309+0000","last_unstale":"2024-03-17T23:55:55.475571+0000","last_undegraded":"2024-03-17T23:55:55.475571+0000","last_fullsized":"2024-03-17T23:55:55.475571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:38:42.130689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.468378+0000","last_change":"2024-03-17T23:54:56.347736+0000","last_active":"2024-03-17T23:55:55.468378+0000","last_peered":"2024-03-17T23:55:55.468378+0000","last_clean":"2024-03-17T23:55:55.468378+0000","last_became_active":"2024-03-17T23:54:56.347283+0000","last_became_peered":"2024-03-17T23:54:56.347283+0000","last_unstale":"2024-03-17T23:55:55.468378+0000","last_undegraded":"2024-03-17T23:55:55.468378+0000","last_fullsized":"2024-03-17T23:55:55.468378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:35:00.683852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.468285+0000","last_change":"2024-03-17T23:54:56.344574+0000","last_active":"2024-03-17T23:55:55.468285+0000","last_peered":"2024-03-17T23:55:55.468285+0000","last_clean":"2024-03-17T23:55:55.468285+0000","last_became_active":"2024-03-17T23:54:56.344272+0000","last_became_peered":"2024-03-17T23:54:56.344272+0000","last_unstale":"2024-03-17T23:55:55.468285+0000","last_undegraded":"2024-03-17T23:55:55.468285+0000","last_fullsized":"2024-03-17T23:55:55.468285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:37:37.134152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.483462+0000","last_change":"2024-03-17T23:54:56.365633+0000","last_active":"2024-03-17T23:55:55.483462+0000","last_peered":"2024-03-17T23:55:55.483462+0000","last_clean":"2024-03-17T23:55:55.483462+0000","last_became_active":"2024-03-17T23:54:56.364533+0000","last_became_peered":"2024-03-17T23:54:56.364533+0000","last_unstale":"2024-03-17T23:55:55.483462+0000","last_undegraded":"2024-03-17T23:55:55.483462+0000","last_fullsized":"2024-03-17T23:55:55.483462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:05:20.848672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.470661+0000","last_change":"2024-03-17T23:54:56.365614+0000","last_active":"2024-03-17T23:55:55.470661+0000","last_peered":"2024-03-17T23:55:55.470661+0000","last_clean":"2024-03-17T23:55:55.470661+0000","last_became_active":"2024-03-17T23:54:56.365301+0000","last_became_peered":"2024-03-17T23:54:56.365301+0000","last_unstale":"2024-03-17T23:55:55.470661+0000","last_undegraded":"2024-03-17T23:55:55.470661+0000","last_fullsized":"2024-03-17T23:55:55.470661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:20.122964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.481102+0000","last_change":"2024-03-17T23:54:56.365531+0000","last_active":"2024-03-17T23:55:55.481102+0000","last_peered":"2024-03-17T23:55:55.481102+0000","last_clean":"2024-03-17T23:55:55.481102+0000","last_became_active":"2024-03-17T23:54:56.364675+0000","last_became_peered":"2024-03-17T23:54:56.364675+0000","last_unstale":"2024-03-17T23:55:55.481102+0000","last_undegraded":"2024-03-17T23:55:55.481102+0000","last_fullsized":"2024-03-17T23:55:55.481102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:25:49.612055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:56:54.118820+0000","last_change":"2024-03-17T23:54:56.355306+0000","last_active":"2024-03-17T23:56:54.118820+0000","last_peered":"2024-03-17T23:56:54.118820+0000","last_clean":"2024-03-17T23:56:54.118820+0000","last_became_active":"2024-03-17T23:54:56.354975+0000","last_became_peered":"2024-03-17T23:54:56.354975+0000","last_unstale":"2024-03-17T23:56:54.118820+0000","last_undegraded":"2024-03-17T23:56:54.118820+0000","last_fullsized":"2024-03-17T23:56:54.118820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:58.606691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.477810+0000","last_change":"2024-03-17T23:54:56.350333+0000","last_active":"2024-03-17T23:55:55.477810+0000","last_peered":"2024-03-17T23:55:55.477810+0000","last_clean":"2024-03-17T23:55:55.477810+0000","last_became_active":"2024-03-17T23:54:56.349953+0000","last_became_peered":"2024-03-17T23:54:56.349953+0000","last_unstale":"2024-03-17T23:55:55.477810+0000","last_undegraded":"2024-03-17T23:55:55.477810+0000","last_fullsized":"2024-03-17T23:55:55.477810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:19:34.639852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-17T23:55:55.466946+0000","last_change":"2024-03-17T23:54:56.343143+0000","last_active":"2024-03-17T23:55:55.466946+0000","last_peered":"2024-03-17T23:55:55.466946+0000","last_clean":"2024-03-17T23:55:55.466946+0000","last_became_active":"2024-03-17T23:54:56.342772+0000","last_became_peered":"2024-03-17T23:54:56.342772+0000","last_unstale":"2024-03-17T23:55:55.466946+0000","last_undegraded":"2024-03-17T23:55:55.466946+0000","last_fullsized":"2024-03-17T23:55:55.466946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:54:55.805681+0000","last_clean_scrub_stamp":"2024-03-17T23:54:55.805681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:33:45.668975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":202,"ondisk_log_size":202,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246647,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71268,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":906,"num_read_kb":533,"num_write":249,"num_write_kb":145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":292,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":213555,"data_compressed":5505,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":509792,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1163264,"data_stored":1160883,"data_compressed":58192,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":513638,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070519,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":509808,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":1106417,"data_compressed":56357,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":505046,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":570531,"data_compressed":22487,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":506502,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":570713,"data_compressed":22487,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":504754,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":509232,"data_compressed":20652,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":1101999,"data_compressed":56357,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34061,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34061,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35705,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":66207,"data_compressed":1835,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66389,"data_compressed":1835,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":66191,"data_compressed":1835,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:56:57.303 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-17T23:56:57.304 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-17T23:56:57.304 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-17T23:56:57.304 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-17T23:56:57.814 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:56:58.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:57 smithi079 ceph-mon[36241]: pgmap v385: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-17T23:56:58.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:57 smithi079 ceph-mon[36241]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:56:58.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:57 smithi191 ceph-mon[36708]: pgmap v385: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-17T23:56:58.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:57 smithi191 ceph-mon[36708]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:56:58.167 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:57 smithi012 ceph-mon[31541]: pgmap v385: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-17T23:56:58.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:57 smithi012 ceph-mon[31541]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-17T23:56:59.835 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:56:59 smithi012 ceph-mon[31541]: pgmap v386: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-17T23:57:00.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:56:59 smithi079 ceph-mon[36241]: pgmap v386: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-17T23:57:00.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:56:59 smithi191 ceph-mon[36708]: pgmap v386: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-17T23:57:00.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.168+0000 7f360e400700 1 -- 172.21.15.12:0/142529834 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3608104ec0 msgr2=0x7f3608111380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.168+0000 7f360e400700 1 --2- 172.21.15.12:0/142529834 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3608104ec0 0x7f3608111380 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f360000b0d0 tx=0x7f360000b3e0 comp rx=0 tx=0).stop 2024-03-17T23:57:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.169+0000 7f360e400700 1 -- 172.21.15.12:0/142529834 shutdown_connections 2024-03-17T23:57:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.169+0000 7f360e400700 1 --2- 172.21.15.12:0/142529834 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3608104ec0 0x7f3608111380 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.169+0000 7f360e400700 1 --2- 172.21.15.12:0/142529834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f360810a5b0 0x7f360810a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.169+0000 7f360e400700 1 --2- 172.21.15.12:0/142529834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3608104500 0x7f3608104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.169+0000 7f360e400700 1 -- 172.21.15.12:0/142529834 >> 172.21.15.12:0/142529834 conn(0x7f36080fffd0 msgr2=0x7f36081023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.169+0000 7f360e400700 1 -- 172.21.15.12:0/142529834 shutdown_connections 2024-03-17T23:57:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.169+0000 7f360e400700 1 -- 172.21.15.12:0/142529834 wait complete. 2024-03-17T23:57:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.170+0000 7f360e400700 1 Processor -- start 2024-03-17T23:57:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.170+0000 7f360e400700 1 -- start start 2024-03-17T23:57:00.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.170+0000 7f360e400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3608104500 0x7f36081a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:00.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.170+0000 7f360e400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3608104ec0 0x7f36081a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:00.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.170+0000 7f360e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f360810a5b0 0x7f360819aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:00.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.170+0000 7f360e400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3608114840 con 0x7f360810a5b0 2024-03-17T23:57:00.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.170+0000 7f360e400700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f36081146c0 con 0x7f3608104500 2024-03-17T23:57:00.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.170+0000 7f360e400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f36081149c0 con 0x7f3608104ec0 2024-03-17T23:57:00.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.171+0000 7f3607400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3608104500 0x7f36081a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:00.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.171+0000 7f3607400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3608104500 0x7f36081a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:39690/0 (socket says 172.21.15.12:39690) 2024-03-17T23:57:00.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.171+0000 7f3607400700 1 -- 172.21.15.12:0/2775836943 learned_addr learned my addr 172.21.15.12:0/2775836943 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:00.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.171+0000 7f35ffe00700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3608104ec0 0x7f36081a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:00.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.171+0000 7f3607e00700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f360810a5b0 0x7f360819aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:00.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.171+0000 7f3607400700 1 -- 172.21.15.12:0/2775836943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3608104ec0 msgr2=0x7f36081a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:00.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.171+0000 7f3607400700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3608104ec0 0x7f36081a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:00.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.171+0000 7f3607400700 1 -- 172.21.15.12:0/2775836943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f360810a5b0 msgr2=0x7f360819aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:00.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.171+0000 7f3607400700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f360810a5b0 0x7f360819aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:00.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.171+0000 7f3607400700 1 -- 172.21.15.12:0/2775836943 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f360819b180 con 0x7f3608104500 2024-03-17T23:57:00.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.171+0000 7f35ffe00700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3608104ec0 0x7f36081a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:00.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.171+0000 7f3607e00700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f360810a5b0 0x7f360819aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:00.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.171+0000 7f3607400700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3608104500 0x7f36081a0810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f35f800e9a0 tx=0x7f35f800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:00.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.172+0000 7f3604c00700 1 -- 172.21.15.12:0/2775836943 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f800cbd0 con 0x7f3608104500 2024-03-17T23:57:00.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.172+0000 7f3604c00700 1 -- 172.21.15.12:0/2775836943 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35f800cd70 con 0x7f3608104500 2024-03-17T23:57:00.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.172+0000 7f360e400700 1 -- 172.21.15.12:0/2775836943 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f360819b470 con 0x7f3608104500 2024-03-17T23:57:00.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.172+0000 7f3604c00700 1 -- 172.21.15.12:0/2775836943 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f8018750 con 0x7f3608104500 2024-03-17T23:57:00.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.172+0000 7f360e400700 1 -- 172.21.15.12:0/2775836943 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36081a76e0 con 0x7f3608104500 2024-03-17T23:57:00.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.173+0000 7f360e400700 1 -- 172.21.15.12:0/2775836943 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3608068f90 con 0x7f3608104500 2024-03-17T23:57:00.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.175+0000 7f3604c00700 1 -- 172.21.15.12:0/2775836943 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f35f8010430 con 0x7f3608104500 2024-03-17T23:57:00.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.176+0000 7f3604c00700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f35ec0a2890 0x7f35ec0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:00.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.177+0000 7f3604c00700 1 -- 172.21.15.12:0/2775836943 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f35f80cf090 con 0x7f3608104500 2024-03-17T23:57:00.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.177+0000 7f35ffe00700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f35ec0a2890 0x7f35ec0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:00.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.177+0000 7f35ffe00700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f35ec0a2890 0x7f35ec0a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3608109b70 tx=0x7f35f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:00.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.183+0000 7f3604c00700 1 -- 172.21.15.12:0/2775836943 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f35f809bd50 con 0x7f3608104500 2024-03-17T23:57:00.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.488+0000 7f360e400700 1 -- 172.21.15.12:0/2775836943 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f360802db70 con 0x7f3608104500 2024-03-17T23:57:00.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.489+0000 7f3604c00700 1 -- 172.21.15.12:0/2775836943 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f35f809b650 con 0x7f3608104500 2024-03-17T23:57:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:00.490 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-17T23:57:00.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.491+0000 7f360e400700 1 -- 172.21.15.12:0/2775836943 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f35ec0a2890 msgr2=0x7f35ec0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:00.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.491+0000 7f360e400700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f35ec0a2890 0x7f35ec0a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3608109b70 tx=0x7f35f0009260 comp rx=0 tx=0).stop 2024-03-17T23:57:00.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.491+0000 7f360e400700 1 -- 172.21.15.12:0/2775836943 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3608104500 msgr2=0x7f36081a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:00.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.491+0000 7f360e400700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3608104500 0x7f36081a0810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f35f800e9a0 tx=0x7f35f800ecb0 comp rx=0 tx=0).stop 2024-03-17T23:57:00.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.492+0000 7f360e400700 1 -- 172.21.15.12:0/2775836943 shutdown_connections 2024-03-17T23:57:00.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.492+0000 7f360e400700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f35ec0a2890 0x7f35ec0a4d50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:00.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.492+0000 7f360e400700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f360810a5b0 0x7f360819aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:00.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.492+0000 7f360e400700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3608104500 0x7f36081a0810 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:00.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.492+0000 7f360e400700 1 --2- 172.21.15.12:0/2775836943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3608104ec0 0x7f36081a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:00.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.492+0000 7f360e400700 1 -- 172.21.15.12:0/2775836943 >> 172.21.15.12:0/2775836943 conn(0x7f36080fffd0 msgr2=0x7f3608101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:00.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.492+0000 7f360e400700 1 -- 172.21.15.12:0/2775836943 shutdown_connections 2024-03-17T23:57:00.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:00.492+0000 7f360e400700 1 -- 172.21.15.12:0/2775836943 wait complete. 2024-03-17T23:57:01.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:00 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2775836943' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-17T23:57:01.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:00 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2775836943' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-17T23:57:01.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:00 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2775836943' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-17T23:57:01.365 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-17T23:57:01.365 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:57:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.602+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3182829299 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2fe0101150 msgr2=0x7f2fe01015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.602+0000 7f2fe5a00640 1 --2- 172.21.15.12:0/3182829299 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2fe0101150 0x7f2fe01015d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8009ca0 tx=0x7f2fd802c0f0 comp rx=0 tx=0).stop 2024-03-17T23:57:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.602+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3182829299 shutdown_connections 2024-03-17T23:57:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.602+0000 7f2fe5a00640 1 --2- 172.21.15.12:0/3182829299 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2fe0101150 0x7f2fe01015d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.602+0000 7f2fe5a00640 1 --2- 172.21.15.12:0/3182829299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fe01007b0 0x7f2fe0100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.602+0000 7f2fe5a00640 1 --2- 172.21.15.12:0/3182829299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fe00ffe50 0x7f2fe0100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.602+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3182829299 >> 172.21.15.12:0/3182829299 conn(0x7f2fe00fba50 msgr2=0x7f2fe00fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.602+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3182829299 shutdown_connections 2024-03-17T23:57:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.602+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3182829299 wait complete. 2024-03-17T23:57:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.603+0000 7f2fe5a00640 1 Processor -- start 2024-03-17T23:57:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.603+0000 7f2fe5a00640 1 -- start start 2024-03-17T23:57:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fe00ffe50 0x7f2fe0199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fe01007b0 0x7f2fe0199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2fe0101150 0x7f2fe01a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe010f430 con 0x7f2fe00ffe50 2024-03-17T23:57:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe010f2b0 con 0x7f2fe0101150 2024-03-17T23:57:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe010f5b0 con 0x7f2fe01007b0 2024-03-17T23:57:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2fe0101150 0x7f2fe01a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fdfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fe00ffe50 0x7f2fe0199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fdf400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fe01007b0 0x7f2fe0199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2fe0101150 0x7f2fe01a0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:39696/0 (socket says 172.21.15.12:39696) 2024-03-17T23:57:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5000640 1 -- 172.21.15.12:0/3641854786 learned_addr learned my addr 172.21.15.12:0/3641854786 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5000640 1 -- 172.21.15.12:0/3641854786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fe01007b0 msgr2=0x7f2fe0199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5000640 1 --2- 172.21.15.12:0/3641854786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fe01007b0 0x7f2fe0199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5000640 1 -- 172.21.15.12:0/3641854786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fe00ffe50 msgr2=0x7f2fe0199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5000640 1 --2- 172.21.15.12:0/3641854786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fe00ffe50 0x7f2fe0199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.604+0000 7f2fe5000640 1 -- 172.21.15.12:0/3641854786 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe01a1660 con 0x7f2fe0101150 2024-03-17T23:57:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.605+0000 7f2fe5000640 1 --2- 172.21.15.12:0/3641854786 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2fe0101150 0x7f2fe01a0ff0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8037040 tx=0x7f2fd8032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.605+0000 7f2fdcc00640 1 -- 172.21.15.12:0/3641854786 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8041020 con 0x7f2fe0101150 2024-03-17T23:57:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.605+0000 7f2fdcc00640 1 -- 172.21.15.12:0/3641854786 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fd8034cd0 con 0x7f2fe0101150 2024-03-17T23:57:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.605+0000 7f2fdcc00640 1 -- 172.21.15.12:0/3641854786 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8039040 con 0x7f2fe0101150 2024-03-17T23:57:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.605+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3641854786 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fe01a18f0 con 0x7f2fe0101150 2024-03-17T23:57:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.605+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3641854786 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fe01a1de0 con 0x7f2fe0101150 2024-03-17T23:57:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.606+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3641854786 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fe010ace0 con 0x7f2fe0101150 2024-03-17T23:57:01.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.608+0000 7f2fdcc00640 1 -- 172.21.15.12:0/3641854786 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2fd8044020 con 0x7f2fe0101150 2024-03-17T23:57:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.610+0000 7f2fdcc00640 1 --2- 172.21.15.12:0/3641854786 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2fc40a2800 0x7f2fc40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.610+0000 7f2fdcc00640 1 -- 172.21.15.12:0/3641854786 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f2fd80f0ce0 con 0x7f2fe0101150 2024-03-17T23:57:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.610+0000 7f2fdfe00640 1 --2- 172.21.15.12:0/3641854786 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2fc40a2800 0x7f2fc40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:01.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.611+0000 7f2fdfe00640 1 --2- 172.21.15.12:0/3641854786 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2fc40a2800 0x7f2fc40a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009bd0 tx=0x7f2fd0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:01.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.614+0000 7f2fdcc00640 1 -- 172.21.15.12:0/3641854786 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2fd804b020 con 0x7f2fe0101150 2024-03-17T23:57:01.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.819+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3641854786 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2fe00623e0 con 0x7f2fe0101150 2024-03-17T23:57:01.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.821+0000 7f2fdcc00640 1 -- 172.21.15.12:0/3641854786 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24618 (secure 0 0 0) 0x7f2fd80bda30 con 0x7f2fe0101150 2024-03-17T23:57:01.822 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:01.825 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":94,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:56:55.719277+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:57:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.824+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3641854786 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2fc40a2800 msgr2=0x7f2fc40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.824+0000 7f2fe5a00640 1 --2- 172.21.15.12:0/3641854786 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2fc40a2800 0x7f2fc40a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009bd0 tx=0x7f2fd0005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.824+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3641854786 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2fe0101150 msgr2=0x7f2fe01a0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.824+0000 7f2fe5a00640 1 --2- 172.21.15.12:0/3641854786 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2fe0101150 0x7f2fe01a0ff0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8037040 tx=0x7f2fd8032f40 comp rx=0 tx=0).stop 2024-03-17T23:57:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.825+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3641854786 shutdown_connections 2024-03-17T23:57:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.825+0000 7f2fe5a00640 1 --2- 172.21.15.12:0/3641854786 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2fc40a2800 0x7f2fc40a4cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:01.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.825+0000 7f2fe5a00640 1 --2- 172.21.15.12:0/3641854786 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2fe0101150 0x7f2fe01a0ff0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:01.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.825+0000 7f2fe5a00640 1 --2- 172.21.15.12:0/3641854786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fe01007b0 0x7f2fe0199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:01.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.825+0000 7f2fe5a00640 1 --2- 172.21.15.12:0/3641854786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fe00ffe50 0x7f2fe0199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:01.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.825+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3641854786 >> 172.21.15.12:0/3641854786 conn(0x7f2fe00fba50 msgr2=0x7f2fe00fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:01.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.825+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3641854786 shutdown_connections 2024-03-17T23:57:01.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:01.825+0000 7f2fe5a00640 1 -- 172.21.15.12:0/3641854786 wait complete. 2024-03-17T23:57:01.844 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:49:38.640282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:54:54.540345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:54:55.803926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:55:00.355066+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:57:01.845 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:57:02.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:01 smithi079 ceph-mon[36241]: pgmap v387: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-17T23:57:02.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.057+0000 7f5a54a00640 1 -- 172.21.15.12:0/82427661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501016e0 msgr2=0x7f5a50101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:02.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.057+0000 7f5a54a00640 1 --2- 172.21.15.12:0/82427661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501016e0 0x7f5a50101b00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5a40009900 tx=0x7f5a4002bc30 comp rx=0 tx=0).stop 2024-03-17T23:57:02.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.057+0000 7f5a54a00640 1 -- 172.21.15.12:0/82427661 shutdown_connections 2024-03-17T23:57:02.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.057+0000 7f5a54a00640 1 --2- 172.21.15.12:0/82427661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a50103290 0x7f5a50109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.057+0000 7f5a54a00640 1 --2- 172.21.15.12:0/82427661 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a501028d0 0x7f5a50102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.057+0000 7f5a54a00640 1 --2- 172.21.15.12:0/82427661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501016e0 0x7f5a50101b00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.057+0000 7f5a54a00640 1 -- 172.21.15.12:0/82427661 >> 172.21.15.12:0/82427661 conn(0x7f5a500fcc50 msgr2=0x7f5a500ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:02.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.057+0000 7f5a54a00640 1 -- 172.21.15.12:0/82427661 shutdown_connections 2024-03-17T23:57:02.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.058+0000 7f5a54a00640 1 -- 172.21.15.12:0/82427661 wait complete. 2024-03-17T23:57:02.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.059+0000 7f5a54a00640 1 Processor -- start 2024-03-17T23:57:02.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.059+0000 7f5a54a00640 1 -- start start 2024-03-17T23:57:02.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.059+0000 7f5a54a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501016e0 0x7f5a50199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:02.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a54a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a501028d0 0x7f5a5019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:02.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a54a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a50103290 0x7f5a501a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:02.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a54a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5010d380 con 0x7f5a50103290 2024-03-17T23:57:02.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a54a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5010d200 con 0x7f5a501028d0 2024-03-17T23:57:02.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a54a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5010d500 con 0x7f5a501016e0 2024-03-17T23:57:02.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a47e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a501028d0 0x7f5a5019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:02.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a4e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501016e0 0x7f5a50199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:02.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a4e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501016e0 0x7f5a50199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:36968/0 (socket says 172.21.15.12:36968) 2024-03-17T23:57:02.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a4e800640 1 -- 172.21.15.12:0/3847506335 learned_addr learned my addr 172.21.15.12:0/3847506335 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:02.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a47e00640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a501028d0 0x7f5a5019a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:39714/0 (socket says 172.21.15.12:39714) 2024-03-17T23:57:02.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a4f200640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a50103290 0x7f5a501a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:02.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a4e800640 1 -- 172.21.15.12:0/3847506335 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a501028d0 msgr2=0x7f5a5019a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:02.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a4e800640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a501028d0 0x7f5a5019a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a4e800640 1 -- 172.21.15.12:0/3847506335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a50103290 msgr2=0x7f5a501a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:02.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a4e800640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a50103290 0x7f5a501a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.060+0000 7f5a4e800640 1 -- 172.21.15.12:0/3847506335 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a501a1800 con 0x7f5a501016e0 2024-03-17T23:57:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.061+0000 7f5a4e800640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501016e0 0x7f5a50199be0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5a40000c00 tx=0x7f5a4002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.061+0000 7f5a47400640 1 -- 172.21.15.12:0/3847506335 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a40043020 con 0x7f5a501016e0 2024-03-17T23:57:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.061+0000 7f5a47400640 1 -- 172.21.15.12:0/3847506335 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a400054e0 con 0x7f5a501016e0 2024-03-17T23:57:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.061+0000 7f5a47400640 1 -- 172.21.15.12:0/3847506335 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a40003c40 con 0x7f5a501016e0 2024-03-17T23:57:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.061+0000 7f5a54a00640 1 -- 172.21.15.12:0/3847506335 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a501a1a90 con 0x7f5a501016e0 2024-03-17T23:57:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.061+0000 7f5a54a00640 1 -- 172.21.15.12:0/3847506335 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a501a1f90 con 0x7f5a501016e0 2024-03-17T23:57:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.062+0000 7f5a54a00640 1 -- 172.21.15.12:0/3847506335 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a501a2160 con 0x7f5a501016e0 2024-03-17T23:57:02.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.064+0000 7f5a47400640 1 -- 172.21.15.12:0/3847506335 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5a40046020 con 0x7f5a501016e0 2024-03-17T23:57:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.065+0000 7f5a47400640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5a340a2880 0x7f5a340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.066+0000 7f5a47400640 1 -- 172.21.15.12:0/3847506335 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f5a400f0da0 con 0x7f5a501016e0 2024-03-17T23:57:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.066+0000 7f5a47e00640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5a340a2880 0x7f5a340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:02.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.067+0000 7f5a47e00640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5a340a2880 0x7f5a340a4d40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5a5019b0f0 tx=0x7f5a38005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:02.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.069+0000 7f5a47400640 1 -- 172.21.15.12:0/3847506335 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5a4004d020 con 0x7f5a501016e0 2024-03-17T23:57:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:01 smithi191 ceph-mon[36708]: pgmap v387: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-17T23:57:02.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:01 smithi012 ceph-mon[31541]: pgmap v387: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-17T23:57:02.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.272+0000 7f5a54a00640 1 -- 172.21.15.12:0/3847506335 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5a501a23d0 con 0x7f5a501016e0 2024-03-17T23:57:02.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.272+0000 7f5a47400640 1 -- 172.21.15.12:0/3847506335 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f5a400bda60 con 0x7f5a501016e0 2024-03-17T23:57:02.273 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-17T23:57:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.275+0000 7f5a54a00640 1 -- 172.21.15.12:0/3847506335 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5a340a2880 msgr2=0x7f5a340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.275+0000 7f5a54a00640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5a340a2880 0x7f5a340a4d40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5a5019b0f0 tx=0x7f5a38005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.275+0000 7f5a54a00640 1 -- 172.21.15.12:0/3847506335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501016e0 msgr2=0x7f5a50199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.275+0000 7f5a54a00640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501016e0 0x7f5a50199be0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5a40000c00 tx=0x7f5a4002c650 comp rx=0 tx=0).stop 2024-03-17T23:57:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.275+0000 7f5a54a00640 1 -- 172.21.15.12:0/3847506335 shutdown_connections 2024-03-17T23:57:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.275+0000 7f5a54a00640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5a340a2880 0x7f5a340a4d40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.275+0000 7f5a54a00640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a50103290 0x7f5a501a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.275+0000 7f5a54a00640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a501028d0 0x7f5a5019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.275+0000 7f5a54a00640 1 --2- 172.21.15.12:0/3847506335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501016e0 0x7f5a50199be0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.275+0000 7f5a54a00640 1 -- 172.21.15.12:0/3847506335 >> 172.21.15.12:0/3847506335 conn(0x7f5a500fcc50 msgr2=0x7f5a50073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:02.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.276+0000 7f5a54a00640 1 -- 172.21.15.12:0/3847506335 shutdown_connections 2024-03-17T23:57:02.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.276+0000 7f5a54a00640 1 -- 172.21.15.12:0/3847506335 wait complete. 2024-03-17T23:57:02.295 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:57:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.523+0000 7f3740800640 1 -- 172.21.15.12:0/2107453388 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3738108ff0 msgr2=0x7f3738109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.523+0000 7f3740800640 1 --2- 172.21.15.12:0/2107453388 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3738108ff0 0x7f3738109470 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3728009900 tx=0x7f372802bc40 comp rx=0 tx=0).stop 2024-03-17T23:57:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.523+0000 7f3740800640 1 -- 172.21.15.12:0/2107453388 shutdown_connections 2024-03-17T23:57:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.523+0000 7f3740800640 1 --2- 172.21.15.12:0/2107453388 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37381099b0 0x7f3738110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.523+0000 7f3740800640 1 --2- 172.21.15.12:0/2107453388 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3738108ff0 0x7f3738109470 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.523+0000 7f3740800640 1 --2- 172.21.15.12:0/2107453388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3738107e00 0x7f3738108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.523+0000 7f3740800640 1 -- 172.21.15.12:0/2107453388 >> 172.21.15.12:0/2107453388 conn(0x7f3738076da0 msgr2=0x7f37380791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:02.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.523+0000 7f3740800640 1 -- 172.21.15.12:0/2107453388 shutdown_connections 2024-03-17T23:57:02.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.523+0000 7f3740800640 1 -- 172.21.15.12:0/2107453388 wait complete. 2024-03-17T23:57:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.524+0000 7f3740800640 1 Processor -- start 2024-03-17T23:57:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.525+0000 7f3740800640 1 -- start start 2024-03-17T23:57:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.525+0000 7f3740800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3738107e00 0x7f373819e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.525+0000 7f3740800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3738108ff0 0x7f373819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.525+0000 7f3740800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37381099b0 0x7f37381a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.525+0000 7f3740800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3738113a60 con 0x7f3738107e00 2024-03-17T23:57:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.525+0000 7f3740800640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f37381138e0 con 0x7f3738108ff0 2024-03-17T23:57:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.525+0000 7f3740800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3738113be0 con 0x7f37381099b0 2024-03-17T23:57:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.525+0000 7f373d400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3738108ff0 0x7f373819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.525+0000 7f373e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37381099b0 0x7f37381a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.525+0000 7f373de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3738107e00 0x7f373819e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.525+0000 7f373d400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3738108ff0 0x7f373819e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:39724/0 (socket says 172.21.15.12:39724) 2024-03-17T23:57:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.525+0000 7f373d400640 1 -- 172.21.15.12:0/4164176586 learned_addr learned my addr 172.21.15.12:0/4164176586 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.526+0000 7f373d400640 1 -- 172.21.15.12:0/4164176586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37381099b0 msgr2=0x7f37381a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.526+0000 7f373d400640 1 --2- 172.21.15.12:0/4164176586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37381099b0 0x7f37381a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.526+0000 7f373d400640 1 -- 172.21.15.12:0/4164176586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3738107e00 msgr2=0x7f373819e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.526+0000 7f373d400640 1 --2- 172.21.15.12:0/4164176586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3738107e00 0x7f373819e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.526+0000 7f373d400640 1 -- 172.21.15.12:0/4164176586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37381a5d20 con 0x7f3738108ff0 2024-03-17T23:57:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.526+0000 7f373d400640 1 --2- 172.21.15.12:0/4164176586 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3738108ff0 0x7f373819e640 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3728009230 tx=0x7f372802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.526+0000 7f3732a00640 1 -- 172.21.15.12:0/4164176586 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728041070 con 0x7f3738108ff0 2024-03-17T23:57:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.526+0000 7f3732a00640 1 -- 172.21.15.12:0/4164176586 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3728004450 con 0x7f3738108ff0 2024-03-17T23:57:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.526+0000 7f3740800640 1 -- 172.21.15.12:0/4164176586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37381a5fb0 con 0x7f3738108ff0 2024-03-17T23:57:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.526+0000 7f3732a00640 1 -- 172.21.15.12:0/4164176586 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728003e00 con 0x7f3738108ff0 2024-03-17T23:57:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.526+0000 7f3740800640 1 -- 172.21.15.12:0/4164176586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37381a6420 con 0x7f3738108ff0 2024-03-17T23:57:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.527+0000 7f3740800640 1 -- 172.21.15.12:0/4164176586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f373810f120 con 0x7f3738108ff0 2024-03-17T23:57:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.529+0000 7f3732a00640 1 -- 172.21.15.12:0/4164176586 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3728044070 con 0x7f3738108ff0 2024-03-17T23:57:02.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.531+0000 7f3732a00640 1 --2- 172.21.15.12:0/4164176586 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f37200a2880 0x7f37200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:02.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.531+0000 7f3732a00640 1 -- 172.21.15.12:0/4164176586 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f37280f0960 con 0x7f3738108ff0 2024-03-17T23:57:02.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.531+0000 7f373de00640 1 --2- 172.21.15.12:0/4164176586 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f37200a2880 0x7f37200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:02.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.532+0000 7f373de00640 1 --2- 172.21.15.12:0/4164176586 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f37200a2880 0x7f37200a4d40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3738108e50 tx=0x7f372c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:02.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.539+0000 7f3732a00640 1 -- 172.21.15.12:0/4164176586 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f37280bcfd0 con 0x7f3738108ff0 2024-03-17T23:57:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.756+0000 7f3740800640 1 -- 172.21.15.12:0/4164176586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3738062480 con 0x7f3738108ff0 2024-03-17T23:57:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.756+0000 7f3732a00640 1 -- 172.21.15.12:0/4164176586 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f37280c1070 con 0x7f3738108ff0 2024-03-17T23:57:02.757 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-17T23:57:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.760+0000 7f3740800640 1 -- 172.21.15.12:0/4164176586 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f37200a2880 msgr2=0x7f37200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.760+0000 7f3740800640 1 --2- 172.21.15.12:0/4164176586 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f37200a2880 0x7f37200a4d40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3738108e50 tx=0x7f372c005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.760+0000 7f3740800640 1 -- 172.21.15.12:0/4164176586 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3738108ff0 msgr2=0x7f373819e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.760+0000 7f3740800640 1 --2- 172.21.15.12:0/4164176586 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3738108ff0 0x7f373819e640 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3728009230 tx=0x7f372802c660 comp rx=0 tx=0).stop 2024-03-17T23:57:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.760+0000 7f3740800640 1 -- 172.21.15.12:0/4164176586 shutdown_connections 2024-03-17T23:57:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.760+0000 7f3740800640 1 --2- 172.21.15.12:0/4164176586 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f37200a2880 0x7f37200a4d40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.760+0000 7f3740800640 1 --2- 172.21.15.12:0/4164176586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37381099b0 0x7f37381a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.760+0000 7f3740800640 1 --2- 172.21.15.12:0/4164176586 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3738108ff0 0x7f373819e640 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.760+0000 7f3740800640 1 --2- 172.21.15.12:0/4164176586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3738107e00 0x7f373819e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.760+0000 7f3740800640 1 -- 172.21.15.12:0/4164176586 >> 172.21.15.12:0/4164176586 conn(0x7f3738076da0 msgr2=0x7f373810c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.760+0000 7f3740800640 1 -- 172.21.15.12:0/4164176586 shutdown_connections 2024-03-17T23:57:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.760+0000 7f3740800640 1 -- 172.21.15.12:0/4164176586 wait complete. 2024-03-17T23:57:02.781 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:57:03.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.999+0000 7f9ea4a00640 1 -- 172.21.15.12:0/2331711833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea01006f0 msgr2=0x7f9ea0100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:03.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.999+0000 7f9ea4a00640 1 --2- 172.21.15.12:0/2331711833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea01006f0 0x7f9ea0100b10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009ca0 tx=0x7f9e9002c0f0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.999+0000 7f9ea4a00640 1 -- 172.21.15.12:0/2331711833 shutdown_connections 2024-03-17T23:57:03.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.999+0000 7f9ea4a00640 1 --2- 172.21.15.12:0/2331711833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ea01022a0 0x7f9ea0108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.999+0000 7f9ea4a00640 1 --2- 172.21.15.12:0/2331711833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea01018e0 0x7f9ea0101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.999+0000 7f9ea4a00640 1 --2- 172.21.15.12:0/2331711833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea01006f0 0x7f9ea0100b10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:02.999+0000 7f9ea4a00640 1 -- 172.21.15.12:0/2331711833 >> 172.21.15.12:0/2331711833 conn(0x7f9ea00fbc80 msgr2=0x7f9ea00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:03.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.000+0000 7f9ea4a00640 1 -- 172.21.15.12:0/2331711833 shutdown_connections 2024-03-17T23:57:03.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.000+0000 7f9ea4a00640 1 -- 172.21.15.12:0/2331711833 wait complete. 2024-03-17T23:57:03.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.000+0000 7f9ea4a00640 1 Processor -- start 2024-03-17T23:57:03.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.001+0000 7f9ea4a00640 1 -- start start 2024-03-17T23:57:03.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.001+0000 7f9ea4a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea01006f0 0x7f9ea0199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:03.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.001+0000 7f9ea4a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ea01018e0 0x7f9ea019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.001+0000 7f9ea4a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea01022a0 0x7f9ea01a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.001+0000 7f9ea4a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea010d4a0 con 0x7f9ea01006f0 2024-03-17T23:57:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.001+0000 7f9ea4a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea010d320 con 0x7f9ea01022a0 2024-03-17T23:57:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.001+0000 7f9ea4a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea010d620 con 0x7f9ea01018e0 2024-03-17T23:57:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.001+0000 7f9e9e800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea01022a0 0x7f9ea01a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.001+0000 7f9e9d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ea01018e0 0x7f9ea019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.001+0000 7f9e9e800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea01022a0 0x7f9ea01a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:39750/0 (socket says 172.21.15.12:39750) 2024-03-17T23:57:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.001+0000 7f9e9e800640 1 -- 172.21.15.12:0/609634723 learned_addr learned my addr 172.21.15.12:0/609634723 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.001+0000 7f9e9de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea01006f0 0x7f9ea0199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.002+0000 7f9e9d400640 1 -- 172.21.15.12:0/609634723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea01022a0 msgr2=0x7f9ea01a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.002+0000 7f9e9d400640 1 --2- 172.21.15.12:0/609634723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea01022a0 0x7f9ea01a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.002+0000 7f9e9d400640 1 -- 172.21.15.12:0/609634723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea01006f0 msgr2=0x7f9ea0199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.002+0000 7f9e9d400640 1 --2- 172.21.15.12:0/609634723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea01006f0 0x7f9ea0199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.002+0000 7f9e9d400640 1 -- 172.21.15.12:0/609634723 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea01a1990 con 0x7f9ea01018e0 2024-03-17T23:57:03.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.002+0000 7f9e9e800640 1 --2- 172.21.15.12:0/609634723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea01022a0 0x7f9ea01a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:03.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.002+0000 7f9e9d400640 1 --2- 172.21.15.12:0/609634723 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ea01018e0 0x7f9ea019a2b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c005fd0 tx=0x7f9e8c00dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:03.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.002+0000 7f9e96a00640 1 -- 172.21.15.12:0/609634723 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c009c30 con 0x7f9ea01018e0 2024-03-17T23:57:03.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.002+0000 7f9e96a00640 1 -- 172.21.15.12:0/609634723 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e8c009dd0 con 0x7f9ea01018e0 2024-03-17T23:57:03.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.002+0000 7f9e96a00640 1 -- 172.21.15.12:0/609634723 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c010460 con 0x7f9ea01018e0 2024-03-17T23:57:03.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.002+0000 7f9ea4a00640 1 -- 172.21.15.12:0/609634723 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea01a1c80 con 0x7f9ea01018e0 2024-03-17T23:57:03.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.003+0000 7f9ea4a00640 1 -- 172.21.15.12:0/609634723 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ea01a2180 con 0x7f9ea01018e0 2024-03-17T23:57:03.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.004+0000 7f9ea4a00640 1 -- 172.21.15.12:0/609634723 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ea01a2350 con 0x7f9ea01018e0 2024-03-17T23:57:03.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.006+0000 7f9e96a00640 1 -- 172.21.15.12:0/609634723 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9e8c003f50 con 0x7f9ea01018e0 2024-03-17T23:57:03.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.007+0000 7f9e96a00640 1 --2- 172.21.15.12:0/609634723 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9e840a2880 0x7f9e840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:03.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.008+0000 7f9e96a00640 1 -- 172.21.15.12:0/609634723 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f9e8c0ced70 con 0x7f9ea01018e0 2024-03-17T23:57:03.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.008+0000 7f9e9de00640 1 --2- 172.21.15.12:0/609634723 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9e840a2880 0x7f9e840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:03.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.009+0000 7f9e9de00640 1 --2- 172.21.15.12:0/609634723 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9e840a2880 0x7f9e840a4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9e9002c630 tx=0x7f9e90009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:03.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.012+0000 7f9e96a00640 1 -- 172.21.15.12:0/609634723 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9e8c017020 con 0x7f9ea01018e0 2024-03-17T23:57:03.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:02 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3641854786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:03.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:02 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3847506335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:57:03.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:02 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/4164176586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:57:03.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:02 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3641854786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:03.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:02 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3847506335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:57:03.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:02 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/4164176586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:57:03.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:02 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3641854786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:03.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:02 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3847506335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:57:03.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:02 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4164176586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:57:03.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.214+0000 7f9ea4a00640 1 -- 172.21.15.12:0/609634723 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9ea01a26b0 con 0x7f9ea01018e0 2024-03-17T23:57:03.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.215+0000 7f9e96a00640 1 -- 172.21.15.12:0/609634723 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f9e8c09b3e0 con 0x7f9ea01018e0 2024-03-17T23:57:03.215 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-17T23:57:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.218+0000 7f9ea4a00640 1 -- 172.21.15.12:0/609634723 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9e840a2880 msgr2=0x7f9e840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.218+0000 7f9ea4a00640 1 --2- 172.21.15.12:0/609634723 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9e840a2880 0x7f9e840a4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9e9002c630 tx=0x7f9e90009360 comp rx=0 tx=0).stop 2024-03-17T23:57:03.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.218+0000 7f9ea4a00640 1 -- 172.21.15.12:0/609634723 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ea01018e0 msgr2=0x7f9ea019a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.218+0000 7f9ea4a00640 1 --2- 172.21.15.12:0/609634723 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ea01018e0 0x7f9ea019a2b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c005fd0 tx=0x7f9e8c00dde0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.219+0000 7f9ea4a00640 1 -- 172.21.15.12:0/609634723 shutdown_connections 2024-03-17T23:57:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.219+0000 7f9ea4a00640 1 --2- 172.21.15.12:0/609634723 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9e840a2880 0x7f9e840a4d40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.219+0000 7f9ea4a00640 1 --2- 172.21.15.12:0/609634723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ea01022a0 0x7f9ea01a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.219+0000 7f9ea4a00640 1 --2- 172.21.15.12:0/609634723 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ea01018e0 0x7f9ea019a2b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.219+0000 7f9ea4a00640 1 --2- 172.21.15.12:0/609634723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea01006f0 0x7f9ea0199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.219+0000 7f9ea4a00640 1 -- 172.21.15.12:0/609634723 >> 172.21.15.12:0/609634723 conn(0x7f9ea00fbc80 msgr2=0x7f9ea01094f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.219+0000 7f9ea4a00640 1 -- 172.21.15.12:0/609634723 shutdown_connections 2024-03-17T23:57:03.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.219+0000 7f9ea4a00640 1 -- 172.21.15.12:0/609634723 wait complete. 2024-03-17T23:57:03.238 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:57:03.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.501+0000 7fb3c3200640 1 -- 172.21.15.12:0/978486644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3c41073a0 msgr2=0x7fb3c41097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:03.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.501+0000 7fb3c3200640 1 --2- 172.21.15.12:0/978486644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3c41073a0 0x7fb3c41097f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc009900 tx=0x7fb3bc02bc30 comp rx=0 tx=0).stop 2024-03-17T23:57:03.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.501+0000 7fb3c3200640 1 -- 172.21.15.12:0/978486644 shutdown_connections 2024-03-17T23:57:03.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.501+0000 7fb3c3200640 1 --2- 172.21.15.12:0/978486644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3c41073a0 0x7fb3c41097f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.501+0000 7fb3c3200640 1 --2- 172.21.15.12:0/978486644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3c4104a70 0x7fb3c4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.501+0000 7fb3c3200640 1 --2- 172.21.15.12:0/978486644 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3c4102140 0x7fb3c4104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.501+0000 7fb3c3200640 1 -- 172.21.15.12:0/978486644 >> 172.21.15.12:0/978486644 conn(0x7fb3c40fbb50 msgr2=0x7fb3c40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:03.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.501+0000 7fb3c3200640 1 -- 172.21.15.12:0/978486644 shutdown_connections 2024-03-17T23:57:03.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.501+0000 7fb3c3200640 1 -- 172.21.15.12:0/978486644 wait complete. 2024-03-17T23:57:03.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.503+0000 7fb3c3200640 1 Processor -- start 2024-03-17T23:57:03.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.503+0000 7fb3c3200640 1 -- start start 2024-03-17T23:57:03.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3c3200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3c4102140 0x7fb3c4199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:03.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3c3200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3c4104a70 0x7fb3c419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:03.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3c3200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3c41073a0 0x7fb3c41a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:03.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3c3200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c410d390 con 0x7fb3c41073a0 2024-03-17T23:57:03.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3c3200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c410d210 con 0x7fb3c4104a70 2024-03-17T23:57:03.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3c3200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c410d510 con 0x7fb3c4102140 2024-03-17T23:57:03.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3c1e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3c4102140 0x7fb3c4199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:03.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3b9400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3c4104a70 0x7fb3c419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:03.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3c1e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3c4102140 0x7fb3c4199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:37024/0 (socket says 172.21.15.12:37024) 2024-03-17T23:57:03.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3b9400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3c4104a70 0x7fb3c419a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:39772/0 (socket says 172.21.15.12:39772) 2024-03-17T23:57:03.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3c1e00640 1 -- 172.21.15.12:0/4250634409 learned_addr learned my addr 172.21.15.12:0/4250634409 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:03.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3c2800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3c41073a0 0x7fb3c41a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:03.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3b9400640 1 -- 172.21.15.12:0/4250634409 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3c4102140 msgr2=0x7fb3c4199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:03.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3b9400640 1 --2- 172.21.15.12:0/4250634409 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3c4102140 0x7fb3c4199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3b9400640 1 -- 172.21.15.12:0/4250634409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3c41073a0 msgr2=0x7fb3c41a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:03.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3b9400640 1 --2- 172.21.15.12:0/4250634409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3c41073a0 0x7fb3c41a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.504+0000 7fb3b9400640 1 -- 172.21.15.12:0/4250634409 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c41a17a0 con 0x7fb3c4104a70 2024-03-17T23:57:03.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.505+0000 7fb3b9400640 1 --2- 172.21.15.12:0/4250634409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3c4104a70 0x7fb3c419a0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00e9c0 tx=0x7fb3ac00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:03.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.505+0000 7fb3bb400640 1 -- 172.21.15.12:0/4250634409 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac00ccc0 con 0x7fb3c4104a70 2024-03-17T23:57:03.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.505+0000 7fb3bb400640 1 -- 172.21.15.12:0/4250634409 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3ac010910 con 0x7fb3c4104a70 2024-03-17T23:57:03.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.505+0000 7fb3c3200640 1 -- 172.21.15.12:0/4250634409 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3c41a1a90 con 0x7fb3c4104a70 2024-03-17T23:57:03.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.505+0000 7fb3bb400640 1 -- 172.21.15.12:0/4250634409 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac0188c0 con 0x7fb3c4104a70 2024-03-17T23:57:03.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.505+0000 7fb3c3200640 1 -- 172.21.15.12:0/4250634409 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3c41a1fe0 con 0x7fb3c4104a70 2024-03-17T23:57:03.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.506+0000 7fb3c3200640 1 -- 172.21.15.12:0/4250634409 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3c4108e90 con 0x7fb3c4104a70 2024-03-17T23:57:03.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.508+0000 7fb3bb400640 1 -- 172.21.15.12:0/4250634409 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb3ac010ac0 con 0x7fb3c4104a70 2024-03-17T23:57:03.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.509+0000 7fb3bb400640 1 --2- 172.21.15.12:0/4250634409 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb3a80a2800 0x7fb3a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:03.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.510+0000 7fb3bb400640 1 -- 172.21.15.12:0/4250634409 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fb3ac0cf1f0 con 0x7fb3c4104a70 2024-03-17T23:57:03.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.510+0000 7fb3c1e00640 1 --2- 172.21.15.12:0/4250634409 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb3a80a2800 0x7fb3a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:03.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.511+0000 7fb3c1e00640 1 --2- 172.21.15.12:0/4250634409 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb3a80a2800 0x7fb3a80a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4009fd0 tx=0x7fb3b4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:03.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.518+0000 7fb3bb400640 1 -- 172.21.15.12:0/4250634409 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb3ac09b860 con 0x7fb3c4104a70 2024-03-17T23:57:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.768+0000 7fb3c3200640 1 -- 172.21.15.12:0/4250634409 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb3c40623e0 con 0x7fb3c4104a70 2024-03-17T23:57:03.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.774+0000 7fb3bb400640 1 -- 172.21.15.12:0/4250634409 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fb3ac09f900 con 0x7fb3c4104a70 2024-03-17T23:57:03.775 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-17T23:57:03.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.778+0000 7fb3c3200640 1 -- 172.21.15.12:0/4250634409 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb3a80a2800 msgr2=0x7fb3a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:03.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.778+0000 7fb3c3200640 1 --2- 172.21.15.12:0/4250634409 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb3a80a2800 0x7fb3a80a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4009fd0 tx=0x7fb3b4009500 comp rx=0 tx=0).stop 2024-03-17T23:57:03.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.778+0000 7fb3c3200640 1 -- 172.21.15.12:0/4250634409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3c4104a70 msgr2=0x7fb3c419a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:03.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.778+0000 7fb3c3200640 1 --2- 172.21.15.12:0/4250634409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3c4104a70 0x7fb3c419a0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00e9c0 tx=0x7fb3ac00ee80 comp rx=0 tx=0).stop 2024-03-17T23:57:03.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.778+0000 7fb3c3200640 1 -- 172.21.15.12:0/4250634409 shutdown_connections 2024-03-17T23:57:03.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.778+0000 7fb3c3200640 1 --2- 172.21.15.12:0/4250634409 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb3a80a2800 0x7fb3a80a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.778+0000 7fb3c3200640 1 --2- 172.21.15.12:0/4250634409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3c41073a0 0x7fb3c41a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.778+0000 7fb3c3200640 1 --2- 172.21.15.12:0/4250634409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb3c4104a70 0x7fb3c419a0c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.778+0000 7fb3c3200640 1 --2- 172.21.15.12:0/4250634409 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3c4102140 0x7fb3c4199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:03.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.778+0000 7fb3c3200640 1 -- 172.21.15.12:0/4250634409 >> 172.21.15.12:0/4250634409 conn(0x7fb3c40fbb50 msgr2=0x7fb3c41017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:03.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.778+0000 7fb3c3200640 1 -- 172.21.15.12:0/4250634409 shutdown_connections 2024-03-17T23:57:03.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:03.779+0000 7fb3c3200640 1 -- 172.21.15.12:0/4250634409 wait complete. 2024-03-17T23:57:03.799 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:57:04.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.014+0000 7f50b3a00640 1 -- 172.21.15.12:0/2652823027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50ac0757a0 msgr2=0x7f50ac073bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:04.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.014+0000 7f50b3a00640 1 --2- 172.21.15.12:0/2652823027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50ac0757a0 0x7f50ac073bf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f50a0009900 tx=0x7f50a002bcc0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.014+0000 7f50b3a00640 1 -- 172.21.15.12:0/2652823027 shutdown_connections 2024-03-17T23:57:04.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.014+0000 7f50b3a00640 1 --2- 172.21.15.12:0/2652823027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50ac1096a0 0x7f50ac10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.014+0000 7f50b3a00640 1 --2- 172.21.15.12:0/2652823027 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50ac0741c0 0x7f50ac074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.014+0000 7f50b3a00640 1 --2- 172.21.15.12:0/2652823027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50ac0757a0 0x7f50ac073bf0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.014+0000 7f50b3a00640 1 -- 172.21.15.12:0/2652823027 >> 172.21.15.12:0/2652823027 conn(0x7f50ac0fcd80 msgr2=0x7f50ac0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:04.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.014+0000 7f50b3a00640 1 -- 172.21.15.12:0/2652823027 shutdown_connections 2024-03-17T23:57:04.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.014+0000 7f50b3a00640 1 -- 172.21.15.12:0/2652823027 wait complete. 2024-03-17T23:57:04.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.015+0000 7f50b3a00640 1 Processor -- start 2024-03-17T23:57:04.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.015+0000 7f50b3a00640 1 -- start start 2024-03-17T23:57:04.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50b3a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50ac0741c0 0x7f50ac199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:04.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50b3a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50ac0757a0 0x7f50ac19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:04.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50b3a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50ac1096a0 0x7f50ac1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:04.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50b3a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f50ac10f6a0 con 0x7f50ac1096a0 2024-03-17T23:57:04.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50b3a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f50ac10f520 con 0x7f50ac0741c0 2024-03-17T23:57:04.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50b3a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f50ac10f820 con 0x7f50ac0757a0 2024-03-17T23:57:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50b1000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50ac0741c0 0x7f50ac199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50b1000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50ac0741c0 0x7f50ac199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56644/0 (socket says 172.21.15.12:56644) 2024-03-17T23:57:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50b1a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50ac1096a0 0x7f50ac1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50b1000640 1 -- 172.21.15.12:0/3813764237 learned_addr learned my addr 172.21.15.12:0/3813764237 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50abe00640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50ac0757a0 0x7f50ac19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:04.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50b1000640 1 -- 172.21.15.12:0/3813764237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50ac0757a0 msgr2=0x7f50ac19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50b1000640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50ac0757a0 0x7f50ac19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.016+0000 7f50b1000640 1 -- 172.21.15.12:0/3813764237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50ac1096a0 msgr2=0x7f50ac1a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.017+0000 7f50b1000640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50ac1096a0 0x7f50ac1a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.017+0000 7f50b1000640 1 -- 172.21.15.12:0/3813764237 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ac1a1900 con 0x7f50ac0741c0 2024-03-17T23:57:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.017+0000 7f50abe00640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50ac0757a0 0x7f50ac19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.017+0000 7f50b1000640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50ac0741c0 0x7f50ac199cc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f50a0005ab0 tx=0x7f50a002c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.017+0000 7f50a9600640 1 -- 172.21.15.12:0/3813764237 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a0043020 con 0x7f50ac0741c0 2024-03-17T23:57:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.017+0000 7f50a9600640 1 -- 172.21.15.12:0/3813764237 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50a00054e0 con 0x7f50ac0741c0 2024-03-17T23:57:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.017+0000 7f50b3a00640 1 -- 172.21.15.12:0/3813764237 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50ac1a1b90 con 0x7f50ac0741c0 2024-03-17T23:57:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.017+0000 7f50a9600640 1 -- 172.21.15.12:0/3813764237 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a0003ca0 con 0x7f50ac0741c0 2024-03-17T23:57:04.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.017+0000 7f50b3a00640 1 -- 172.21.15.12:0/3813764237 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50ac1a2130 con 0x7f50ac0741c0 2024-03-17T23:57:04.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.019+0000 7f50b3a00640 1 -- 172.21.15.12:0/3813764237 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50ac108460 con 0x7f50ac0741c0 2024-03-17T23:57:04.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.020+0000 7f50a9600640 1 -- 172.21.15.12:0/3813764237 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f50a0046020 con 0x7f50ac0741c0 2024-03-17T23:57:04.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.022+0000 7f50a9600640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f50940a2880 0x7f50940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:04.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.022+0000 7f50a9600640 1 -- 172.21.15.12:0/3813764237 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f50a00f0f70 con 0x7f50ac0741c0 2024-03-17T23:57:04.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.022+0000 7f50abe00640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f50940a2880 0x7f50940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:04.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.023+0000 7f50abe00640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f50940a2880 0x7f50940a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f50ac19b1c0 tx=0x7f5098005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:04.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.026+0000 7f50a9600640 1 -- 172.21.15.12:0/3813764237 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f50a004d020 con 0x7f50ac0741c0 2024-03-17T23:57:04.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:03 smithi079 ceph-mon[36241]: pgmap v388: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-17T23:57:04.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:03 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/609634723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:57:04.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:03 smithi191 ceph-mon[36708]: pgmap v388: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-17T23:57:04.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:03 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/609634723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:57:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:03 smithi012 ceph-mon[31541]: pgmap v388: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-17T23:57:04.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:03 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/609634723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:57:04.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.270+0000 7f50b3a00640 1 -- 172.21.15.12:0/3813764237 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50ac0623e0 con 0x7f50ac0741c0 2024-03-17T23:57:04.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.271+0000 7f50a9600640 1 -- 172.21.15.12:0/3813764237 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11936 (secure 0 0 0) 0x7f50a00bdd50 con 0x7f50ac0741c0 2024-03-17T23:57:04.272 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:04.273 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-17T23:54:56.312554+0000","modified":"2024-03-17T23:55:28.030481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24575,"mds_2":24530,"mds_3":24529,"mds_4":24514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.12:6841/2073607248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":2073607248},{"type":"v1","addr":"172.21.15.12:6841","nonce":2073607248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/2478742711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":2478742711},{"type":"v1","addr":"172.21.15.79:6839","nonce":2478742711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24575,24587,24530,24529,24514]},"id":1}]} 2024-03-17T23:57:04.273 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 28 2024-03-17T23:57:04.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.274+0000 7f50b3a00640 1 -- 172.21.15.12:0/3813764237 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f50940a2880 msgr2=0x7f50940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:04.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.274+0000 7f50b3a00640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f50940a2880 0x7f50940a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f50ac19b1c0 tx=0x7f5098005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:04.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.274+0000 7f50b3a00640 1 -- 172.21.15.12:0/3813764237 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50ac0741c0 msgr2=0x7f50ac199cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:04.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.274+0000 7f50b3a00640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50ac0741c0 0x7f50ac199cc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f50a0005ab0 tx=0x7f50a002c750 comp rx=0 tx=0).stop 2024-03-17T23:57:04.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.274+0000 7f50b3a00640 1 -- 172.21.15.12:0/3813764237 shutdown_connections 2024-03-17T23:57:04.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.274+0000 7f50b3a00640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f50940a2880 0x7f50940a4d40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.274+0000 7f50b3a00640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50ac1096a0 0x7f50ac1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.274+0000 7f50b3a00640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50ac0757a0 0x7f50ac19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.274+0000 7f50b3a00640 1 --2- 172.21.15.12:0/3813764237 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50ac0741c0 0x7f50ac199cc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.274+0000 7f50b3a00640 1 -- 172.21.15.12:0/3813764237 >> 172.21.15.12:0/3813764237 conn(0x7f50ac0fcd80 msgr2=0x7f50ac105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:04.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.274+0000 7f50b3a00640 1 -- 172.21.15.12:0/3813764237 shutdown_connections 2024-03-17T23:57:04.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.275+0000 7f50b3a00640 1 -- 172.21.15.12:0/3813764237 wait complete. 2024-03-17T23:57:04.295 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:57:04.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.502+0000 7f2ccc600640 1 -- 172.21.15.12:0/3639730403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cc4104bf0 msgr2=0x7f2cc4105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:04.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.502+0000 7f2ccc600640 1 --2- 172.21.15.12:0/3639730403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cc4104bf0 0x7f2cc4105070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009900 tx=0x7f2cb002bc30 comp rx=0 tx=0).stop 2024-03-17T23:57:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.502+0000 7f2ccc600640 1 -- 172.21.15.12:0/3639730403 shutdown_connections 2024-03-17T23:57:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.502+0000 7f2ccc600640 1 --2- 172.21.15.12:0/3639730403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2cc41055b0 0x7f2cc410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.502+0000 7f2ccc600640 1 --2- 172.21.15.12:0/3639730403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cc4104bf0 0x7f2cc4105070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.502+0000 7f2ccc600640 1 --2- 172.21.15.12:0/3639730403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103a00 0x7f2cc4103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.502+0000 7f2ccc600640 1 -- 172.21.15.12:0/3639730403 >> 172.21.15.12:0/3639730403 conn(0x7f2cc40fef90 msgr2=0x7f2cc41013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.502+0000 7f2ccc600640 1 -- 172.21.15.12:0/3639730403 shutdown_connections 2024-03-17T23:57:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.502+0000 7f2ccc600640 1 -- 172.21.15.12:0/3639730403 wait complete. 2024-03-17T23:57:04.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.503+0000 7f2ccc600640 1 Processor -- start 2024-03-17T23:57:04.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.504+0000 7f2ccc600640 1 -- start start 2024-03-17T23:57:04.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.504+0000 7f2ccc600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103a00 0x7f2cc4199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.504+0000 7f2ccc600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cc4104bf0 0x7f2cc419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.504+0000 7f2ccc600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2cc41055b0 0x7f2cc41a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.504+0000 7f2ccc600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc410f680 con 0x7f2cc4103a00 2024-03-17T23:57:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.504+0000 7f2ccc600640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc410f500 con 0x7f2cc41055b0 2024-03-17T23:57:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.504+0000 7f2ccc600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc410f800 con 0x7f2cc4104bf0 2024-03-17T23:57:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.504+0000 7f2cca600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2cc41055b0 0x7f2cc41a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.504+0000 7f2cc9200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cc4104bf0 0x7f2cc419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.504+0000 7f2cc9c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103a00 0x7f2cc4199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.504+0000 7f2cca600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2cc41055b0 0x7f2cc41a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56656/0 (socket says 172.21.15.12:56656) 2024-03-17T23:57:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.504+0000 7f2cca600640 1 -- 172.21.15.12:0/3902043407 learned_addr learned my addr 172.21.15.12:0/3902043407 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.505+0000 7f2cca600640 1 -- 172.21.15.12:0/3902043407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cc4104bf0 msgr2=0x7f2cc419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.505+0000 7f2cca600640 1 --2- 172.21.15.12:0/3902043407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cc4104bf0 0x7f2cc419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.505+0000 7f2cca600640 1 -- 172.21.15.12:0/3902043407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103a00 msgr2=0x7f2cc4199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.505+0000 7f2cca600640 1 --2- 172.21.15.12:0/3902043407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103a00 0x7f2cc4199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.505+0000 7f2cca600640 1 -- 172.21.15.12:0/3902043407 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc41a1920 con 0x7f2cc41055b0 2024-03-17T23:57:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.505+0000 7f2cc9200640 1 --2- 172.21.15.12:0/3902043407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cc4104bf0 0x7f2cc419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.505+0000 7f2cca600640 1 --2- 172.21.15.12:0/3902043407 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2cc41055b0 0x7f2cc41a1280 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00eb10 tx=0x7f2cbc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.505+0000 7f2cc2000640 1 -- 172.21.15.12:0/3902043407 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc004b20 con 0x7f2cc41055b0 2024-03-17T23:57:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.505+0000 7f2cc2000640 1 -- 172.21.15.12:0/3902043407 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cbc004cc0 con 0x7f2cc41055b0 2024-03-17T23:57:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.505+0000 7f2cc2000640 1 -- 172.21.15.12:0/3902043407 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc010760 con 0x7f2cc41055b0 2024-03-17T23:57:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.505+0000 7f2ccc600640 1 -- 172.21.15.12:0/3902043407 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc41a1c10 con 0x7f2cc41055b0 2024-03-17T23:57:04.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.506+0000 7f2ccc600640 1 -- 172.21.15.12:0/3902043407 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc41a2110 con 0x7f2cc41055b0 2024-03-17T23:57:04.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.507+0000 7f2ccc600640 1 -- 172.21.15.12:0/3902043407 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cc410ad20 con 0x7f2cc41055b0 2024-03-17T23:57:04.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.509+0000 7f2cc2000640 1 -- 172.21.15.12:0/3902043407 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2cbc005020 con 0x7f2cc41055b0 2024-03-17T23:57:04.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.510+0000 7f2cc2000640 1 --2- 172.21.15.12:0/3902043407 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2cac0a2800 0x7f2cac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:04.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.510+0000 7f2cc2000640 1 -- 172.21.15.12:0/3902043407 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f2cbc0cf590 con 0x7f2cc41055b0 2024-03-17T23:57:04.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.510+0000 7f2cc9c00640 1 --2- 172.21.15.12:0/3902043407 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2cac0a2800 0x7f2cac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:04.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.511+0000 7f2cc9c00640 1 --2- 172.21.15.12:0/3902043407 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2cac0a2800 0x7f2cac0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2cc419b1e0 tx=0x7f2cb8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:04.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.514+0000 7f2cc2000640 1 -- 172.21.15.12:0/3902043407 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2cbc09c250 con 0x7f2cc41055b0 2024-03-17T23:57:04.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.720+0000 7f2ccc600640 1 -- 172.21.15.12:0/3902043407 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2cc402d430 con 0x7f2cc41055b0 2024-03-17T23:57:04.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.722+0000 7f2cc2000640 1 -- 172.21.15.12:0/3902043407 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24618 (secure 0 0 0) 0x7f2cbc09bb50 con 0x7f2cc41055b0 2024-03-17T23:57:04.723 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:04.723 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":94,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:56:55.719277+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:57:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.724+0000 7f2ccc600640 1 -- 172.21.15.12:0/3902043407 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2cac0a2800 msgr2=0x7f2cac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.724+0000 7f2ccc600640 1 --2- 172.21.15.12:0/3902043407 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2cac0a2800 0x7f2cac0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2cc419b1e0 tx=0x7f2cb8005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.724+0000 7f2ccc600640 1 -- 172.21.15.12:0/3902043407 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2cc41055b0 msgr2=0x7f2cc41a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.724+0000 7f2ccc600640 1 --2- 172.21.15.12:0/3902043407 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2cc41055b0 0x7f2cc41a1280 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00eb10 tx=0x7f2cbc00c2d0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.725+0000 7f2ccc600640 1 -- 172.21.15.12:0/3902043407 shutdown_connections 2024-03-17T23:57:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.725+0000 7f2ccc600640 1 --2- 172.21.15.12:0/3902043407 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2cac0a2800 0x7f2cac0a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.725+0000 7f2ccc600640 1 --2- 172.21.15.12:0/3902043407 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2cc41055b0 0x7f2cc41a1280 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.725+0000 7f2ccc600640 1 --2- 172.21.15.12:0/3902043407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cc4104bf0 0x7f2cc419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.725+0000 7f2ccc600640 1 --2- 172.21.15.12:0/3902043407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103a00 0x7f2cc4199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.725+0000 7f2ccc600640 1 -- 172.21.15.12:0/3902043407 >> 172.21.15.12:0/3902043407 conn(0x7f2cc40fef90 msgr2=0x7f2cc4107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.725+0000 7f2ccc600640 1 -- 172.21.15.12:0/3902043407 shutdown_connections 2024-03-17T23:57:04.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.725+0000 7f2ccc600640 1 -- 172.21.15.12:0/3902043407 wait complete. 2024-03-17T23:57:04.747 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:49:38.640282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:54:54.540345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:54:55.803926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:55:00.355066+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:57:04.747 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:57:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.974+0000 7f33d9200640 1 -- 172.21.15.12:0/828953517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33d41028d0 msgr2=0x7f33d4102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:04.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.974+0000 7f33d9200640 1 --2- 172.21.15.12:0/828953517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33d41028d0 0x7f33d4102d50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009900 tx=0x7f33c002bc40 comp rx=0 tx=0).stop 2024-03-17T23:57:04.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.974+0000 7f33d9200640 1 -- 172.21.15.12:0/828953517 shutdown_connections 2024-03-17T23:57:04.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.974+0000 7f33d9200640 1 --2- 172.21.15.12:0/828953517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d4103290 0x7f33d4109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.974+0000 7f33d9200640 1 --2- 172.21.15.12:0/828953517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33d41028d0 0x7f33d4102d50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.974+0000 7f33d9200640 1 --2- 172.21.15.12:0/828953517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33d41016e0 0x7f33d4101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.974+0000 7f33d9200640 1 -- 172.21.15.12:0/828953517 >> 172.21.15.12:0/828953517 conn(0x7f33d40fcc50 msgr2=0x7f33d40ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:04.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.974+0000 7f33d9200640 1 -- 172.21.15.12:0/828953517 shutdown_connections 2024-03-17T23:57:04.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.975+0000 7f33d9200640 1 -- 172.21.15.12:0/828953517 wait complete. 2024-03-17T23:57:04.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.975+0000 7f33d9200640 1 Processor -- start 2024-03-17T23:57:04.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.976+0000 7f33d9200640 1 -- start start 2024-03-17T23:57:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.976+0000 7f33d9200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d41016e0 0x7f33d4075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.976+0000 7f33d9200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33d41028d0 0x7f33d4073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.976+0000 7f33d9200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33d4103290 0x7f33d4074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.976+0000 7f33d9200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f33d410d3b0 con 0x7f33d41016e0 2024-03-17T23:57:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.976+0000 7f33d9200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f33d410d230 con 0x7f33d41028d0 2024-03-17T23:57:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.976+0000 7f33d9200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f33d410d530 con 0x7f33d4103290 2024-03-17T23:57:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.976+0000 7f33d3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33d4103290 0x7f33d4074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.976+0000 7f33d3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d41016e0 0x7f33d4075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.976+0000 7f33d2a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33d41028d0 0x7f33d4073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.976+0000 7f33d3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33d4103290 0x7f33d4074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53360/0 (socket says 172.21.15.12:53360) 2024-03-17T23:57:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.976+0000 7f33d3e00640 1 -- 172.21.15.12:0/3311174181 learned_addr learned my addr 172.21.15.12:0/3311174181 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.977+0000 7f33d3e00640 1 -- 172.21.15.12:0/3311174181 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33d41028d0 msgr2=0x7f33d4073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:04.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.977+0000 7f33d3e00640 1 --2- 172.21.15.12:0/3311174181 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33d41028d0 0x7f33d4073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.977+0000 7f33d3e00640 1 -- 172.21.15.12:0/3311174181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d41016e0 msgr2=0x7f33d4075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.977+0000 7f33d3e00640 1 --2- 172.21.15.12:0/3311174181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d41016e0 0x7f33d4075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.977+0000 7f33d3e00640 1 -- 172.21.15.12:0/3311174181 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d40749a0 con 0x7f33d4103290 2024-03-17T23:57:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.977+0000 7f33d3e00640 1 --2- 172.21.15.12:0/3311174181 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33d4103290 0x7f33d4074110 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009fd0 tx=0x7f33cc00b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.977+0000 7f33cbe00640 1 -- 172.21.15.12:0/3311174181 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc017070 con 0x7f33d4103290 2024-03-17T23:57:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.977+0000 7f33cbe00640 1 -- 172.21.15.12:0/3311174181 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33cc003e50 con 0x7f33d4103290 2024-03-17T23:57:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.977+0000 7f33cbe00640 1 -- 172.21.15.12:0/3311174181 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc0106c0 con 0x7f33d4103290 2024-03-17T23:57:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.977+0000 7f33d9200640 1 -- 172.21.15.12:0/3311174181 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33d41a1b40 con 0x7f33d4103290 2024-03-17T23:57:04.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.977+0000 7f33d9200640 1 -- 172.21.15.12:0/3311174181 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33d41a2060 con 0x7f33d4103290 2024-03-17T23:57:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.979+0000 7f33d9200640 1 -- 172.21.15.12:0/3311174181 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33d40684f0 con 0x7f33d4103290 2024-03-17T23:57:04.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.980+0000 7f33cbe00640 1 -- 172.21.15.12:0/3311174181 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f33cc003970 con 0x7f33d4103290 2024-03-17T23:57:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.981+0000 7f33cbe00640 1 --2- 172.21.15.12:0/3311174181 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f33b80a2880 0x7f33b80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.982+0000 7f33cbe00640 1 -- 172.21.15.12:0/3311174181 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f33cc0cea30 con 0x7f33d4103290 2024-03-17T23:57:04.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.982+0000 7f33d3400640 1 --2- 172.21.15.12:0/3311174181 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f33b80a2880 0x7f33b80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:04.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.982+0000 7f33d3400640 1 --2- 172.21.15.12:0/3311174181 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f33b80a2880 0x7f33b80a4d40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f33d4102730 tx=0x7f33c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:04.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:04.985+0000 7f33cbe00640 1 -- 172.21.15.12:0/3311174181 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f33cc09b0a0 con 0x7f33d4103290 2024-03-17T23:57:05.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:04 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/4250634409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:57:05.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:04 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3813764237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:05.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:04 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3902043407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:05.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:04 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/4250634409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:57:05.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:04 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3813764237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:05.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:04 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3902043407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:05.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:04 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4250634409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:57:05.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:04 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3813764237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:05.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:04 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3902043407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.190+0000 7f33d9200640 1 -- 172.21.15.12:0/3311174181 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f33d40623e0 con 0x7f33d4103290 2024-03-17T23:57:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.190+0000 7f33cbe00640 1 -- 172.21.15.12:0/3311174181 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f33cc09f140 con 0x7f33d4103290 2024-03-17T23:57:05.191 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-17T23:57:05.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.193+0000 7f33d9200640 1 -- 172.21.15.12:0/3311174181 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f33b80a2880 msgr2=0x7f33b80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:05.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.193+0000 7f33d9200640 1 --2- 172.21.15.12:0/3311174181 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f33b80a2880 0x7f33b80a4d40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f33d4102730 tx=0x7f33c4005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:05.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.193+0000 7f33d9200640 1 -- 172.21.15.12:0/3311174181 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33d4103290 msgr2=0x7f33d4074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:05.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.193+0000 7f33d9200640 1 --2- 172.21.15.12:0/3311174181 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33d4103290 0x7f33d4074110 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009fd0 tx=0x7f33cc00b260 comp rx=0 tx=0).stop 2024-03-17T23:57:05.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.194+0000 7f33d9200640 1 -- 172.21.15.12:0/3311174181 shutdown_connections 2024-03-17T23:57:05.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.194+0000 7f33d9200640 1 --2- 172.21.15.12:0/3311174181 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f33b80a2880 0x7f33b80a4d40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.194+0000 7f33d9200640 1 --2- 172.21.15.12:0/3311174181 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33d4103290 0x7f33d4074110 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.194+0000 7f33d9200640 1 --2- 172.21.15.12:0/3311174181 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f33d41028d0 0x7f33d4073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.194+0000 7f33d9200640 1 --2- 172.21.15.12:0/3311174181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d41016e0 0x7f33d4075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.194+0000 7f33d9200640 1 -- 172.21.15.12:0/3311174181 >> 172.21.15.12:0/3311174181 conn(0x7f33d40fcc50 msgr2=0x7f33d4105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:05.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.194+0000 7f33d9200640 1 -- 172.21.15.12:0/3311174181 shutdown_connections 2024-03-17T23:57:05.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.194+0000 7f33d9200640 1 -- 172.21.15.12:0/3311174181 wait complete. 2024-03-17T23:57:05.215 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:57:05.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.415+0000 7f54e4600640 1 -- 172.21.15.12:0/2714169542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54dc1017f0 msgr2=0x7f54dc101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:05.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.415+0000 7f54e4600640 1 --2- 172.21.15.12:0/2714169542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54dc1017f0 0x7f54dc101c10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009900 tx=0x7f54d002bc30 comp rx=0 tx=0).stop 2024-03-17T23:57:05.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.415+0000 7f54e4600640 1 -- 172.21.15.12:0/2714169542 shutdown_connections 2024-03-17T23:57:05.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.415+0000 7f54e4600640 1 --2- 172.21.15.12:0/2714169542 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54dc1033a0 0x7f54dc109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.415+0000 7f54e4600640 1 --2- 172.21.15.12:0/2714169542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54dc1029e0 0x7f54dc102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.415+0000 7f54e4600640 1 --2- 172.21.15.12:0/2714169542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54dc1017f0 0x7f54dc101c10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.415+0000 7f54e4600640 1 -- 172.21.15.12:0/2714169542 >> 172.21.15.12:0/2714169542 conn(0x7f54dc0fcd80 msgr2=0x7f54dc0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:05.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.415+0000 7f54e4600640 1 -- 172.21.15.12:0/2714169542 shutdown_connections 2024-03-17T23:57:05.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.416+0000 7f54e4600640 1 -- 172.21.15.12:0/2714169542 wait complete. 2024-03-17T23:57:05.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.417+0000 7f54e4600640 1 Processor -- start 2024-03-17T23:57:05.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.417+0000 7f54e4600640 1 -- start start 2024-03-17T23:57:05.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.417+0000 7f54e4600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54dc1017f0 0x7f54dc199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:05.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.417+0000 7f54e4600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54dc1029e0 0x7f54dc19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:05.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.417+0000 7f54e4600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54dc1033a0 0x7f54dc1a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:05.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.417+0000 7f54e4600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc10d490 con 0x7f54dc1029e0 2024-03-17T23:57:05.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.417+0000 7f54e4600640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc10d310 con 0x7f54dc1017f0 2024-03-17T23:57:05.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.417+0000 7f54e4600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc10d610 con 0x7f54dc1033a0 2024-03-17T23:57:05.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.417+0000 7f54e1c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54dc1017f0 0x7f54dc199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:05.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.418+0000 7f54e1c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54dc1017f0 0x7f54dc199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56680/0 (socket says 172.21.15.12:56680) 2024-03-17T23:57:05.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.418+0000 7f54e1c00640 1 -- 172.21.15.12:0/4138086758 learned_addr learned my addr 172.21.15.12:0/4138086758 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:05.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.418+0000 7f54e2600640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54dc1033a0 0x7f54dc1a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:05.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.418+0000 7f54e1200640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54dc1029e0 0x7f54dc19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:05.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.418+0000 7f54e1c00640 1 -- 172.21.15.12:0/4138086758 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54dc1033a0 msgr2=0x7f54dc1a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:05.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.418+0000 7f54e1c00640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54dc1033a0 0x7f54dc1a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.418+0000 7f54e1c00640 1 -- 172.21.15.12:0/4138086758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54dc1029e0 msgr2=0x7f54dc19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:05.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.418+0000 7f54e1c00640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54dc1029e0 0x7f54dc19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.418+0000 7f54e1c00640 1 -- 172.21.15.12:0/4138086758 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54dc1a1970 con 0x7f54dc1017f0 2024-03-17T23:57:05.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.418+0000 7f54e1200640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54dc1029e0 0x7f54dc19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:05.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.418+0000 7f54e1c00640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54dc1017f0 0x7f54dc199d50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f54d0000c00 tx=0x7f54d002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:05.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.419+0000 7f54da000640 1 -- 172.21.15.12:0/4138086758 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0043020 con 0x7f54dc1017f0 2024-03-17T23:57:05.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.419+0000 7f54da000640 1 -- 172.21.15.12:0/4138086758 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54d00054e0 con 0x7f54dc1017f0 2024-03-17T23:57:05.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.419+0000 7f54da000640 1 -- 172.21.15.12:0/4138086758 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0003c40 con 0x7f54dc1017f0 2024-03-17T23:57:05.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.419+0000 7f54e4600640 1 -- 172.21.15.12:0/4138086758 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54dc1a1c00 con 0x7f54dc1017f0 2024-03-17T23:57:05.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.419+0000 7f54e4600640 1 -- 172.21.15.12:0/4138086758 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54dc1a2150 con 0x7f54dc1017f0 2024-03-17T23:57:05.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.420+0000 7f54e4600640 1 -- 172.21.15.12:0/4138086758 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54dc1a2320 con 0x7f54dc1017f0 2024-03-17T23:57:05.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.423+0000 7f54da000640 1 -- 172.21.15.12:0/4138086758 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f54d0046020 con 0x7f54dc1017f0 2024-03-17T23:57:05.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.425+0000 7f54da000640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f54c40a2800 0x7f54c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:05.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.425+0000 7f54da000640 1 -- 172.21.15.12:0/4138086758 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f54d00f0e90 con 0x7f54dc1017f0 2024-03-17T23:57:05.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.425+0000 7f54e1200640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f54c40a2800 0x7f54c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:05.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.426+0000 7f54e1200640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f54c40a2800 0x7f54c40a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f54dc19b260 tx=0x7f54c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:05.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.433+0000 7f54da000640 1 -- 172.21.15.12:0/4138086758 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f54d004d020 con 0x7f54dc1017f0 2024-03-17T23:57:05.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.706+0000 7f54e4600640 1 -- 172.21.15.12:0/4138086758 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f54dc1a24f0 con 0x7f54dc1017f0 2024-03-17T23:57:05.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.706+0000 7f54da000640 1 -- 172.21.15.12:0/4138086758 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f54d00bdb50 con 0x7f54dc1017f0 2024-03-17T23:57:05.707 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-17T23:57:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.709+0000 7f54e4600640 1 -- 172.21.15.12:0/4138086758 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f54c40a2800 msgr2=0x7f54c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:05.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.709+0000 7f54e4600640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f54c40a2800 0x7f54c40a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f54dc19b260 tx=0x7f54c8005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.709+0000 7f54e4600640 1 -- 172.21.15.12:0/4138086758 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54dc1017f0 msgr2=0x7f54dc199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.709+0000 7f54e4600640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54dc1017f0 0x7f54dc199d50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f54d0000c00 tx=0x7f54d002c650 comp rx=0 tx=0).stop 2024-03-17T23:57:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.709+0000 7f54e4600640 1 -- 172.21.15.12:0/4138086758 shutdown_connections 2024-03-17T23:57:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.709+0000 7f54e4600640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f54c40a2800 0x7f54c40a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.709+0000 7f54e4600640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54dc1033a0 0x7f54dc1a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.709+0000 7f54e4600640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54dc1029e0 0x7f54dc19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.709+0000 7f54e4600640 1 --2- 172.21.15.12:0/4138086758 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54dc1017f0 0x7f54dc199d50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.709+0000 7f54e4600640 1 -- 172.21.15.12:0/4138086758 >> 172.21.15.12:0/4138086758 conn(0x7f54dc0fcd80 msgr2=0x7f54dc073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.709+0000 7f54e4600640 1 -- 172.21.15.12:0/4138086758 shutdown_connections 2024-03-17T23:57:05.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.709+0000 7f54e4600640 1 -- 172.21.15.12:0/4138086758 wait complete. 2024-03-17T23:57:05.730 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:57:05.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:05 smithi012 ceph-mon[31541]: pgmap v389: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 542 B/s wr, 11 op/s 2024-03-17T23:57:05.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:05 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3311174181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:57:05.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:05 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4138086758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:57:05.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.925+0000 7f7dce200640 1 -- 172.21.15.12:0/2692014227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dc81016e0 msgr2=0x7f7dc8101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:05.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.925+0000 7f7dce200640 1 --2- 172.21.15.12:0/2692014227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dc81016e0 0x7f7dc8101b00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc009900 tx=0x7f7dbc02bc50 comp rx=0 tx=0).stop 2024-03-17T23:57:05.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.926+0000 7f7dce200640 1 -- 172.21.15.12:0/2692014227 shutdown_connections 2024-03-17T23:57:05.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.926+0000 7f7dce200640 1 --2- 172.21.15.12:0/2692014227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dc8103290 0x7f7dc8109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.926+0000 7f7dce200640 1 --2- 172.21.15.12:0/2692014227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dc81028d0 0x7f7dc8102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.926+0000 7f7dce200640 1 --2- 172.21.15.12:0/2692014227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dc81016e0 0x7f7dc8101b00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.926+0000 7f7dce200640 1 -- 172.21.15.12:0/2692014227 >> 172.21.15.12:0/2692014227 conn(0x7f7dc80fcc50 msgr2=0x7f7dc80ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:05.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.926+0000 7f7dce200640 1 -- 172.21.15.12:0/2692014227 shutdown_connections 2024-03-17T23:57:05.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.926+0000 7f7dce200640 1 -- 172.21.15.12:0/2692014227 wait complete. 2024-03-17T23:57:05.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.927+0000 7f7dce200640 1 Processor -- start 2024-03-17T23:57:05.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.927+0000 7f7dce200640 1 -- start start 2024-03-17T23:57:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.927+0000 7f7dce200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dc81016e0 0x7f7dc8199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.927+0000 7f7dce200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dc81028d0 0x7f7dc819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.927+0000 7f7dce200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dc8103290 0x7f7dc81a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.927+0000 7f7dce200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc810d3a0 con 0x7f7dc81016e0 2024-03-17T23:57:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.927+0000 7f7dce200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc810d220 con 0x7f7dc8103290 2024-03-17T23:57:05.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.927+0000 7f7dce200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc810d520 con 0x7f7dc81028d0 2024-03-17T23:57:05.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.927+0000 7f7dcce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dc81016e0 0x7f7dc8199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:05.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.927+0000 7f7dc7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dc81028d0 0x7f7dc819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:05.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.927+0000 7f7dcd800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dc8103290 0x7f7dc81a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:05.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.928+0000 7f7dcce00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dc81016e0 0x7f7dc8199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46908/0 (socket says 172.21.15.12:46908) 2024-03-17T23:57:05.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.928+0000 7f7dcce00640 1 -- 172.21.15.12:0/2221120375 learned_addr learned my addr 172.21.15.12:0/2221120375 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:05.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.928+0000 7f7dcd800640 1 -- 172.21.15.12:0/2221120375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dc81028d0 msgr2=0x7f7dc819a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:05.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.928+0000 7f7dcd800640 1 --2- 172.21.15.12:0/2221120375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dc81028d0 0x7f7dc819a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.928+0000 7f7dcd800640 1 -- 172.21.15.12:0/2221120375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dc81016e0 msgr2=0x7f7dc8199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:05.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.928+0000 7f7dcd800640 1 --2- 172.21.15.12:0/2221120375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dc81016e0 0x7f7dc8199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:05.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.928+0000 7f7dcd800640 1 -- 172.21.15.12:0/2221120375 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dc81a1800 con 0x7f7dc8103290 2024-03-17T23:57:05.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.928+0000 7f7dcd800640 1 --2- 172.21.15.12:0/2221120375 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dc8103290 0x7f7dc81a1190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0009fd0 tx=0x7f7dc000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:05.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.928+0000 7f7dc5600640 1 -- 172.21.15.12:0/2221120375 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc000ce20 con 0x7f7dc8103290 2024-03-17T23:57:05.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.929+0000 7f7dc5600640 1 -- 172.21.15.12:0/2221120375 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dc0005020 con 0x7f7dc8103290 2024-03-17T23:57:05.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.929+0000 7f7dc5600640 1 -- 172.21.15.12:0/2221120375 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0010720 con 0x7f7dc8103290 2024-03-17T23:57:05.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.929+0000 7f7dce200640 1 -- 172.21.15.12:0/2221120375 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dc81a1af0 con 0x7f7dc8103290 2024-03-17T23:57:05.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.929+0000 7f7dce200640 1 -- 172.21.15.12:0/2221120375 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dc81a2040 con 0x7f7dc8103290 2024-03-17T23:57:05.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.930+0000 7f7dce200640 1 -- 172.21.15.12:0/2221120375 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dc81a2270 con 0x7f7dc8103290 2024-03-17T23:57:05.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.932+0000 7f7dc5600640 1 -- 172.21.15.12:0/2221120375 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7dc0005540 con 0x7f7dc8103290 2024-03-17T23:57:05.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.933+0000 7f7dc5600640 1 --2- 172.21.15.12:0/2221120375 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7db00a2880 0x7f7db00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:05.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.933+0000 7f7dc5600640 1 -- 172.21.15.12:0/2221120375 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f7dc00cf420 con 0x7f7dc8103290 2024-03-17T23:57:05.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.933+0000 7f7dcce00640 1 --2- 172.21.15.12:0/2221120375 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7db00a2880 0x7f7db00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:05.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.934+0000 7f7dcce00640 1 --2- 172.21.15.12:0/2221120375 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7db00a2880 0x7f7db00a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc02c900 tx=0x7f7dbc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:05.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:05.937+0000 7f7dc5600640 1 -- 172.21.15.12:0/2221120375 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7dc009c140 con 0x7f7dc8103290 2024-03-17T23:57:06.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:05 smithi079 ceph-mon[36241]: pgmap v389: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 542 B/s wr, 11 op/s 2024-03-17T23:57:06.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:05 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3311174181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:57:06.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:05 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/4138086758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:57:06.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:05 smithi191 ceph-mon[36708]: pgmap v389: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 542 B/s wr, 11 op/s 2024-03-17T23:57:06.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:05 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3311174181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:57:06.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:05 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/4138086758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:57:06.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.141+0000 7f7dce200640 1 -- 172.21.15.12:0/2221120375 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7dc81a2440 con 0x7f7dc8103290 2024-03-17T23:57:06.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.141+0000 7f7dc5600640 1 -- 172.21.15.12:0/2221120375 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f7dc009ba40 con 0x7f7dc8103290 2024-03-17T23:57:06.142 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-17T23:57:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.144+0000 7f7dce200640 1 -- 172.21.15.12:0/2221120375 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7db00a2880 msgr2=0x7f7db00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.144+0000 7f7dce200640 1 --2- 172.21.15.12:0/2221120375 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7db00a2880 0x7f7db00a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc02c900 tx=0x7f7dbc000bc0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.144+0000 7f7dce200640 1 -- 172.21.15.12:0/2221120375 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dc8103290 msgr2=0x7f7dc81a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.144+0000 7f7dce200640 1 --2- 172.21.15.12:0/2221120375 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dc8103290 0x7f7dc81a1190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0009fd0 tx=0x7f7dc000ef10 comp rx=0 tx=0).stop 2024-03-17T23:57:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.144+0000 7f7dce200640 1 -- 172.21.15.12:0/2221120375 shutdown_connections 2024-03-17T23:57:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.144+0000 7f7dce200640 1 --2- 172.21.15.12:0/2221120375 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f7db00a2880 0x7f7db00a4d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.144+0000 7f7dce200640 1 --2- 172.21.15.12:0/2221120375 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dc8103290 0x7f7dc81a1190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.144+0000 7f7dce200640 1 --2- 172.21.15.12:0/2221120375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dc81028d0 0x7f7dc819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.144+0000 7f7dce200640 1 --2- 172.21.15.12:0/2221120375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7dc81016e0 0x7f7dc8199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.144+0000 7f7dce200640 1 -- 172.21.15.12:0/2221120375 >> 172.21.15.12:0/2221120375 conn(0x7f7dc80fcc50 msgr2=0x7f7dc8073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.144+0000 7f7dce200640 1 -- 172.21.15.12:0/2221120375 shutdown_connections 2024-03-17T23:57:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.144+0000 7f7dce200640 1 -- 172.21.15.12:0/2221120375 wait complete. 2024-03-17T23:57:06.164 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:57:06.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.364+0000 7f86af200640 1 -- 172.21.15.12:0/2010672315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86b01016e0 msgr2=0x7f86b0101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:06.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.364+0000 7f86af200640 1 --2- 172.21.15.12:0/2010672315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86b01016e0 0x7f86b0101b00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f86a0009ca0 tx=0x7f86a002c0f0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.365+0000 7f86af200640 1 -- 172.21.15.12:0/2010672315 shutdown_connections 2024-03-17T23:57:06.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.365+0000 7f86af200640 1 --2- 172.21.15.12:0/2010672315 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86b0103290 0x7f86b0109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.365+0000 7f86af200640 1 --2- 172.21.15.12:0/2010672315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86b01028d0 0x7f86b0102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.365+0000 7f86af200640 1 --2- 172.21.15.12:0/2010672315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86b01016e0 0x7f86b0101b00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.365+0000 7f86af200640 1 -- 172.21.15.12:0/2010672315 >> 172.21.15.12:0/2010672315 conn(0x7f86b00fcc50 msgr2=0x7f86b00ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:06.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.365+0000 7f86af200640 1 -- 172.21.15.12:0/2010672315 shutdown_connections 2024-03-17T23:57:06.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.365+0000 7f86af200640 1 -- 172.21.15.12:0/2010672315 wait complete. 2024-03-17T23:57:06.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.366+0000 7f86af200640 1 Processor -- start 2024-03-17T23:57:06.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.366+0000 7f86af200640 1 -- start start 2024-03-17T23:57:06.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.366+0000 7f86af200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86b01016e0 0x7f86b0199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:06.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.366+0000 7f86af200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86b01028d0 0x7f86b019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:06.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.366+0000 7f86af200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86b0103290 0x7f86b01a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.366+0000 7f86af200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f86b010d330 con 0x7f86b01028d0 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.366+0000 7f86af200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f86b010d1b0 con 0x7f86b0103290 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.366+0000 7f86af200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f86b010d4b0 con 0x7f86b01016e0 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.367+0000 7f86ade00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86b01016e0 0x7f86b0199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.367+0000 7f86ae800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86b0103290 0x7f86b01a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.367+0000 7f86ade00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86b01016e0 0x7f86b0199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53406/0 (socket says 172.21.15.12:53406) 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.367+0000 7f86ae800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86b0103290 0x7f86b01a11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56706/0 (socket says 172.21.15.12:56706) 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.367+0000 7f86ade00640 1 -- 172.21.15.12:0/1878568400 learned_addr learned my addr 172.21.15.12:0/1878568400 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.367+0000 7f86a5400640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86b01028d0 0x7f86b019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.367+0000 7f86ade00640 1 -- 172.21.15.12:0/1878568400 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86b0103290 msgr2=0x7f86b01a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.367+0000 7f86ade00640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86b0103290 0x7f86b01a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.367+0000 7f86ade00640 1 -- 172.21.15.12:0/1878568400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86b01028d0 msgr2=0x7f86b019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.367+0000 7f86ade00640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86b01028d0 0x7f86b019a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.367+0000 7f86ade00640 1 -- 172.21.15.12:0/1878568400 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b01a1870 con 0x7f86b01016e0 2024-03-17T23:57:06.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.367+0000 7f86ae800640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86b0103290 0x7f86b01a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:06.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.367+0000 7f86ade00640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86b01016e0 0x7f86b0199cb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f86a00324c0 tx=0x7f86a0032f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:06.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.368+0000 7f86a7400640 1 -- 172.21.15.12:0/1878568400 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a0041020 con 0x7f86b01016e0 2024-03-17T23:57:06.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.368+0000 7f86a7400640 1 -- 172.21.15.12:0/1878568400 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86a0034cd0 con 0x7f86b01016e0 2024-03-17T23:57:06.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.368+0000 7f86a7400640 1 -- 172.21.15.12:0/1878568400 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a0039040 con 0x7f86b01016e0 2024-03-17T23:57:06.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.368+0000 7f86af200640 1 -- 172.21.15.12:0/1878568400 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86b01a1b00 con 0x7f86b01016e0 2024-03-17T23:57:06.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.368+0000 7f86af200640 1 -- 172.21.15.12:0/1878568400 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86b01a1fa0 con 0x7f86b01016e0 2024-03-17T23:57:06.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.369+0000 7f86af200640 1 -- 172.21.15.12:0/1878568400 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86b01a2170 con 0x7f86b01016e0 2024-03-17T23:57:06.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.371+0000 7f86a7400640 1 -- 172.21.15.12:0/1878568400 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f86a0044020 con 0x7f86b01016e0 2024-03-17T23:57:06.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.372+0000 7f86a7400640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f86940a2880 0x7f86940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:06.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.372+0000 7f86a7400640 1 -- 172.21.15.12:0/1878568400 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f86a00f0da0 con 0x7f86b01016e0 2024-03-17T23:57:06.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.373+0000 7f86a5400640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f86940a2880 0x7f86940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:06.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.373+0000 7f86a5400640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f86940a2880 0x7f86940a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8698009b60 tx=0x7f8698005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.376+0000 7f86a7400640 1 -- 172.21.15.12:0/1878568400 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f86a004b020 con 0x7f86b01016e0 2024-03-17T23:57:06.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.579+0000 7f86af200640 1 -- 172.21.15.12:0/1878568400 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f86b01a28e0 con 0x7f86b01016e0 2024-03-17T23:57:06.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.579+0000 7f86a7400640 1 -- 172.21.15.12:0/1878568400 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f86a00bda60 con 0x7f86b01016e0 2024-03-17T23:57:06.580 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-17T23:57:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.582+0000 7f86af200640 1 -- 172.21.15.12:0/1878568400 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f86940a2880 msgr2=0x7f86940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.582+0000 7f86af200640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f86940a2880 0x7f86940a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8698009b60 tx=0x7f8698005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.582+0000 7f86af200640 1 -- 172.21.15.12:0/1878568400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86b01016e0 msgr2=0x7f86b0199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.582+0000 7f86af200640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86b01016e0 0x7f86b0199cb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f86a00324c0 tx=0x7f86a0032f40 comp rx=0 tx=0).stop 2024-03-17T23:57:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.582+0000 7f86af200640 1 -- 172.21.15.12:0/1878568400 shutdown_connections 2024-03-17T23:57:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.582+0000 7f86af200640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f86940a2880 0x7f86940a4d40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.582+0000 7f86af200640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86b0103290 0x7f86b01a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.582+0000 7f86af200640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86b01028d0 0x7f86b019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.582+0000 7f86af200640 1 --2- 172.21.15.12:0/1878568400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86b01016e0 0x7f86b0199cb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.582+0000 7f86af200640 1 -- 172.21.15.12:0/1878568400 >> 172.21.15.12:0/1878568400 conn(0x7f86b00fcc50 msgr2=0x7f86b0073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.582+0000 7f86af200640 1 -- 172.21.15.12:0/1878568400 shutdown_connections 2024-03-17T23:57:06.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.582+0000 7f86af200640 1 -- 172.21.15.12:0/1878568400 wait complete. 2024-03-17T23:57:06.599 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:57:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.790+0000 7f7335200640 1 -- 172.21.15.12:0/4003015299 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73301017f0 msgr2=0x7f7330101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.790+0000 7f7335200640 1 --2- 172.21.15.12:0/4003015299 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73301017f0 0x7f7330101c10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7320009900 tx=0x7f732002bc30 comp rx=0 tx=0).stop 2024-03-17T23:57:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.790+0000 7f7335200640 1 -- 172.21.15.12:0/4003015299 shutdown_connections 2024-03-17T23:57:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.790+0000 7f7335200640 1 --2- 172.21.15.12:0/4003015299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73301033a0 0x7f7330109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.790+0000 7f7335200640 1 --2- 172.21.15.12:0/4003015299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73301029e0 0x7f7330102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.790+0000 7f7335200640 1 --2- 172.21.15.12:0/4003015299 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73301017f0 0x7f7330101c10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.790+0000 7f7335200640 1 -- 172.21.15.12:0/4003015299 >> 172.21.15.12:0/4003015299 conn(0x7f73300fcd80 msgr2=0x7f73300ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.790+0000 7f7335200640 1 -- 172.21.15.12:0/4003015299 shutdown_connections 2024-03-17T23:57:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.790+0000 7f7335200640 1 -- 172.21.15.12:0/4003015299 wait complete. 2024-03-17T23:57:06.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.791+0000 7f7335200640 1 Processor -- start 2024-03-17T23:57:06.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.791+0000 7f7335200640 1 -- start start 2024-03-17T23:57:06.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.792+0000 7f7335200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73301017f0 0x7f7330075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:06.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.792+0000 7f7335200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73301029e0 0x7f7330073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:06.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.792+0000 7f7335200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73301033a0 0x7f7330074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.792+0000 7f7335200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f733010d470 con 0x7f73301029e0 2024-03-17T23:57:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.792+0000 7f7335200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f733010d2f0 con 0x7f73301017f0 2024-03-17T23:57:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.792+0000 7f7335200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f733010d5f0 con 0x7f73301033a0 2024-03-17T23:57:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.792+0000 7f732de00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73301017f0 0x7f7330075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.792+0000 7f732d400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73301029e0 0x7f7330073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.792+0000 7f732e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73301033a0 0x7f7330074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.792+0000 7f732de00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73301017f0 0x7f7330075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56718/0 (socket says 172.21.15.12:56718) 2024-03-17T23:57:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.792+0000 7f732de00640 1 -- 172.21.15.12:0/2805556287 learned_addr learned my addr 172.21.15.12:0/2805556287 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.793+0000 7f732de00640 1 -- 172.21.15.12:0/2805556287 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73301033a0 msgr2=0x7f7330074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.793+0000 7f732de00640 1 --2- 172.21.15.12:0/2805556287 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73301033a0 0x7f7330074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.793+0000 7f732de00640 1 -- 172.21.15.12:0/2805556287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73301029e0 msgr2=0x7f7330073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.793+0000 7f732de00640 1 --2- 172.21.15.12:0/2805556287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73301029e0 0x7f7330073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.793+0000 7f732de00640 1 -- 172.21.15.12:0/2805556287 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73300749a0 con 0x7f73301017f0 2024-03-17T23:57:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.793+0000 7f732e800640 1 --2- 172.21.15.12:0/2805556287 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73301033a0 0x7f7330074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.793+0000 7f732de00640 1 --2- 172.21.15.12:0/2805556287 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73301017f0 0x7f7330075590 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7320000c00 tx=0x7f7320005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.793+0000 7f7326a00640 1 -- 172.21.15.12:0/2805556287 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320041070 con 0x7f73301017f0 2024-03-17T23:57:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.793+0000 7f7326a00640 1 -- 172.21.15.12:0/2805556287 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73200044d0 con 0x7f73301017f0 2024-03-17T23:57:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.793+0000 7f7326a00640 1 -- 172.21.15.12:0/2805556287 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320034440 con 0x7f73301017f0 2024-03-17T23:57:06.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.793+0000 7f7335200640 1 -- 172.21.15.12:0/2805556287 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73301a1bd0 con 0x7f73301017f0 2024-03-17T23:57:06.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.793+0000 7f7335200640 1 -- 172.21.15.12:0/2805556287 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73301a2170 con 0x7f73301017f0 2024-03-17T23:57:06.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.794+0000 7f7335200640 1 -- 172.21.15.12:0/2805556287 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7330108b10 con 0x7f73301017f0 2024-03-17T23:57:06.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.796+0000 7f7326a00640 1 -- 172.21.15.12:0/2805556287 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7320044070 con 0x7f73301017f0 2024-03-17T23:57:06.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.798+0000 7f7326a00640 1 --2- 172.21.15.12:0/2805556287 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f73140a2880 0x7f73140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:06.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.798+0000 7f7326a00640 1 -- 172.21.15.12:0/2805556287 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f73200f0b00 con 0x7f73301017f0 2024-03-17T23:57:06.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.798+0000 7f732d400640 1 --2- 172.21.15.12:0/2805556287 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f73140a2880 0x7f73140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:06.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.799+0000 7f732d400640 1 --2- 172.21.15.12:0/2805556287 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f73140a2880 0x7f73140a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7330075300 tx=0x7f731c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:06.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:06.800+0000 7f7326a00640 1 -- 172.21.15.12:0/2805556287 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f73200bd170 con 0x7f73301017f0 2024-03-17T23:57:06.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:06 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2221120375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:57:06.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:06 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1878568400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:57:07.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:06 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2221120375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:57:07.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:06 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1878568400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:57:07.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:06 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2221120375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:57:07.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:06 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1878568400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:57:07.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.146+0000 7f7335200640 1 -- 172.21.15.12:0/2805556287 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f733002d1b0 con 0x7f73301017f0 2024-03-17T23:57:07.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.146+0000 7f7326a00640 1 -- 172.21.15.12:0/2805556287 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11936 (secure 0 0 0) 0x7f73200c1210 con 0x7f73301017f0 2024-03-17T23:57:07.147 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:07.148 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-17T23:54:56.312554+0000","modified":"2024-03-17T23:55:28.030481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24575,"mds_2":24530,"mds_3":24529,"mds_4":24514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.12:6841/2073607248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":2073607248},{"type":"v1","addr":"172.21.15.12:6841","nonce":2073607248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/2478742711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":2478742711},{"type":"v1","addr":"172.21.15.79:6839","nonce":2478742711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24575,24587,24530,24529,24514]},"id":1}]} 2024-03-17T23:57:07.148 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 28 2024-03-17T23:57:07.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.149+0000 7f7335200640 1 -- 172.21.15.12:0/2805556287 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f73140a2880 msgr2=0x7f73140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:07.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.149+0000 7f7335200640 1 --2- 172.21.15.12:0/2805556287 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f73140a2880 0x7f73140a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7330075300 tx=0x7f731c00b390 comp rx=0 tx=0).stop 2024-03-17T23:57:07.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.149+0000 7f7335200640 1 -- 172.21.15.12:0/2805556287 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73301017f0 msgr2=0x7f7330075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:07.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.149+0000 7f7335200640 1 --2- 172.21.15.12:0/2805556287 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73301017f0 0x7f7330075590 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7320000c00 tx=0x7f7320005d50 comp rx=0 tx=0).stop 2024-03-17T23:57:07.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.150+0000 7f7335200640 1 -- 172.21.15.12:0/2805556287 shutdown_connections 2024-03-17T23:57:07.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.150+0000 7f7335200640 1 --2- 172.21.15.12:0/2805556287 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f73140a2880 0x7f73140a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.150+0000 7f7335200640 1 --2- 172.21.15.12:0/2805556287 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73301033a0 0x7f7330074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.150+0000 7f7335200640 1 --2- 172.21.15.12:0/2805556287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73301029e0 0x7f7330073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.150+0000 7f7335200640 1 --2- 172.21.15.12:0/2805556287 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f73301017f0 0x7f7330075590 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.150+0000 7f7335200640 1 -- 172.21.15.12:0/2805556287 >> 172.21.15.12:0/2805556287 conn(0x7f73300fcd80 msgr2=0x7f7330105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:07.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.150+0000 7f7335200640 1 -- 172.21.15.12:0/2805556287 shutdown_connections 2024-03-17T23:57:07.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.150+0000 7f7335200640 1 -- 172.21.15.12:0/2805556287 wait complete. 2024-03-17T23:57:07.171 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:57:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.382+0000 7f6f5fa00640 1 -- 172.21.15.12:0/1399361986 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f58100a10 msgr2=0x7f6f58100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.382+0000 7f6f5fa00640 1 --2- 172.21.15.12:0/1399361986 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f58100a10 0x7f6f58100e70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009900 tx=0x7f6f4002bc40 comp rx=0 tx=0).stop 2024-03-17T23:57:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.382+0000 7f6f5fa00640 1 -- 172.21.15.12:0/1399361986 shutdown_connections 2024-03-17T23:57:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.382+0000 7f6f5fa00640 1 --2- 172.21.15.12:0/1399361986 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f581013b0 0x7f6f58101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.382+0000 7f6f5fa00640 1 --2- 172.21.15.12:0/1399361986 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f58100a10 0x7f6f58100e70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.382+0000 7f6f5fa00640 1 --2- 172.21.15.12:0/1399361986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f581000b0 0x7f6f581004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.382+0000 7f6f5fa00640 1 -- 172.21.15.12:0/1399361986 >> 172.21.15.12:0/1399361986 conn(0x7f6f580fbc70 msgr2=0x7f6f580fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.382+0000 7f6f5fa00640 1 -- 172.21.15.12:0/1399361986 shutdown_connections 2024-03-17T23:57:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.382+0000 7f6f5fa00640 1 -- 172.21.15.12:0/1399361986 wait complete. 2024-03-17T23:57:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.383+0000 7f6f5fa00640 1 Processor -- start 2024-03-17T23:57:07.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.383+0000 7f6f5fa00640 1 -- start start 2024-03-17T23:57:07.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5fa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f581000b0 0x7f6f58199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5fa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f58100a10 0x7f6f5819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5fa00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f581013b0 0x7f6f581a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5fa00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5810f610 con 0x7f6f58100a10 2024-03-17T23:57:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5fa00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5810f490 con 0x7f6f581013b0 2024-03-17T23:57:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5fa00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5810f790 con 0x7f6f581000b0 2024-03-17T23:57:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5d000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f581000b0 0x7f6f58199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f4fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f58100a10 0x7f6f5819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5d000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f581000b0 0x7f6f58199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53456/0 (socket says 172.21.15.12:53456) 2024-03-17T23:57:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5da00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f581013b0 0x7f6f581a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f4fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f58100a10 0x7f6f5819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46944/0 (socket says 172.21.15.12:46944) 2024-03-17T23:57:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5d000640 1 -- 172.21.15.12:0/3839273276 learned_addr learned my addr 172.21.15.12:0/3839273276 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5d000640 1 -- 172.21.15.12:0/3839273276 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f581013b0 msgr2=0x7f6f581a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5d000640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f581013b0 0x7f6f581a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5d000640 1 -- 172.21.15.12:0/3839273276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f58100a10 msgr2=0x7f6f5819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5d000640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f58100a10 0x7f6f5819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5d000640 1 -- 172.21.15.12:0/3839273276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f581a18c0 con 0x7f6f581000b0 2024-03-17T23:57:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f5da00640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f581013b0 0x7f6f581a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.384+0000 7f6f4fe00640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f58100a10 0x7f6f5819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.385+0000 7f6f5d000640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f581000b0 0x7f6f58199c80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800ebc0 tx=0x7f6f4800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.385+0000 7f6f56000640 1 -- 172.21.15.12:0/3839273276 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48010630 con 0x7f6f581000b0 2024-03-17T23:57:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.385+0000 7f6f56000640 1 -- 172.21.15.12:0/3839273276 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f48010cb0 con 0x7f6f581000b0 2024-03-17T23:57:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.385+0000 7f6f56000640 1 -- 172.21.15.12:0/3839273276 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48018bd0 con 0x7f6f581000b0 2024-03-17T23:57:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.385+0000 7f6f5fa00640 1 -- 172.21.15.12:0/3839273276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f581a1bb0 con 0x7f6f581000b0 2024-03-17T23:57:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.385+0000 7f6f5fa00640 1 -- 172.21.15.12:0/3839273276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f581a2120 con 0x7f6f581000b0 2024-03-17T23:57:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.386+0000 7f6f5fa00640 1 -- 172.21.15.12:0/3839273276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f5810af90 con 0x7f6f581000b0 2024-03-17T23:57:07.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.388+0000 7f6f56000640 1 -- 172.21.15.12:0/3839273276 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6f480107d0 con 0x7f6f581000b0 2024-03-17T23:57:07.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.389+0000 7f6f56000640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6f3c0a2880 0x7f6f3c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:07.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.390+0000 7f6f56000640 1 -- 172.21.15.12:0/3839273276 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6f480cf3a0 con 0x7f6f581000b0 2024-03-17T23:57:07.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.390+0000 7f6f4fe00640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6f3c0a2880 0x7f6f3c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:07.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.390+0000 7f6f4fe00640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6f3c0a2880 0x7f6f3c0a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6f40000c00 tx=0x7f6f40005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:07.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.398+0000 7f6f56000640 1 -- 172.21.15.12:0/3839273276 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6f4809ba10 con 0x7f6f581000b0 2024-03-17T23:57:07.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.603+0000 7f6f5fa00640 1 -- 172.21.15.12:0/3839273276 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f580623e0 con 0x7f6f581000b0 2024-03-17T23:57:07.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.605+0000 7f6f56000640 1 -- 172.21.15.12:0/3839273276 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24618 (secure 0 0 0) 0x7f6f4809fab0 con 0x7f6f581000b0 2024-03-17T23:57:07.606 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:07.607 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":94,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:56:55.719277+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:57:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.608+0000 7f6f5fa00640 1 -- 172.21.15.12:0/3839273276 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6f3c0a2880 msgr2=0x7f6f3c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.608+0000 7f6f5fa00640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6f3c0a2880 0x7f6f3c0a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6f40000c00 tx=0x7f6f40005fd0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.608+0000 7f6f5fa00640 1 -- 172.21.15.12:0/3839273276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f581000b0 msgr2=0x7f6f58199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.608+0000 7f6f5fa00640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f581000b0 0x7f6f58199c80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800ebc0 tx=0x7f6f4800c6a0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.608+0000 7f6f5fa00640 1 -- 172.21.15.12:0/3839273276 shutdown_connections 2024-03-17T23:57:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.608+0000 7f6f5fa00640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6f3c0a2880 0x7f6f3c0a4d40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.608+0000 7f6f5fa00640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f581013b0 0x7f6f581a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.608+0000 7f6f5fa00640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f58100a10 0x7f6f5819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.608+0000 7f6f5fa00640 1 --2- 172.21.15.12:0/3839273276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f581000b0 0x7f6f58199c80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.608+0000 7f6f5fa00640 1 -- 172.21.15.12:0/3839273276 >> 172.21.15.12:0/3839273276 conn(0x7f6f580fbc70 msgr2=0x7f6f580fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.608+0000 7f6f5fa00640 1 -- 172.21.15.12:0/3839273276 shutdown_connections 2024-03-17T23:57:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.608+0000 7f6f5fa00640 1 -- 172.21.15.12:0/3839273276 wait complete. 2024-03-17T23:57:07.629 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:57:07.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.836+0000 7f2616400640 1 -- 172.21.15.12:0/2304758424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26101029e0 msgr2=0x7f2610102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:07.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.836+0000 7f2616400640 1 --2- 172.21.15.12:0/2304758424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26101029e0 0x7f2610102e60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009900 tx=0x7f25f802bc30 comp rx=0 tx=0).stop 2024-03-17T23:57:07.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.836+0000 7f2616400640 1 -- 172.21.15.12:0/2304758424 shutdown_connections 2024-03-17T23:57:07.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.836+0000 7f2616400640 1 --2- 172.21.15.12:0/2304758424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26101033a0 0x7f2610109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.836+0000 7f2616400640 1 --2- 172.21.15.12:0/2304758424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26101029e0 0x7f2610102e60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.836+0000 7f2616400640 1 --2- 172.21.15.12:0/2304758424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26101017f0 0x7f2610101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.836+0000 7f2616400640 1 -- 172.21.15.12:0/2304758424 >> 172.21.15.12:0/2304758424 conn(0x7f26100fcd80 msgr2=0x7f26100ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:07.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.836+0000 7f2616400640 1 -- 172.21.15.12:0/2304758424 shutdown_connections 2024-03-17T23:57:07.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.836+0000 7f2616400640 1 -- 172.21.15.12:0/2304758424 wait complete. 2024-03-17T23:57:07.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.838+0000 7f2616400640 1 Processor -- start 2024-03-17T23:57:07.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.838+0000 7f2616400640 1 -- start start 2024-03-17T23:57:07.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.838+0000 7f2616400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26101017f0 0x7f2610197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:07.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.838+0000 7f2616400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26101029e0 0x7f2610198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:07.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.838+0000 7f2616400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26101033a0 0x7f261019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:07.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.838+0000 7f2616400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f261010d470 con 0x7f26101017f0 2024-03-17T23:57:07.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.838+0000 7f2616400640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f261010d2f0 con 0x7f26101029e0 2024-03-17T23:57:07.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.838+0000 7f2616400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f261010d5f0 con 0x7f26101033a0 2024-03-17T23:57:07.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.839+0000 7f260f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26101017f0 0x7f2610197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:07.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.839+0000 7f2606a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26101029e0 0x7f2610198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:07.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.839+0000 7f260fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26101033a0 0x7f261019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:07.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.839+0000 7f2606a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26101029e0 0x7f2610198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56746/0 (socket says 172.21.15.12:56746) 2024-03-17T23:57:07.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.839+0000 7f2606a00640 1 -- 172.21.15.12:0/2655532029 learned_addr learned my addr 172.21.15.12:0/2655532029 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:07.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.839+0000 7f2606a00640 1 -- 172.21.15.12:0/2655532029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26101033a0 msgr2=0x7f261019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:07.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.839+0000 7f2606a00640 1 --2- 172.21.15.12:0/2655532029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26101033a0 0x7f261019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.839+0000 7f2606a00640 1 -- 172.21.15.12:0/2655532029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26101017f0 msgr2=0x7f2610197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:07.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.839+0000 7f2606a00640 1 --2- 172.21.15.12:0/2655532029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26101017f0 0x7f2610197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:07.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.839+0000 7f2606a00640 1 -- 172.21.15.12:0/2655532029 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f261019f730 con 0x7f26101029e0 2024-03-17T23:57:07.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.839+0000 7f2606a00640 1 --2- 172.21.15.12:0/2655532029 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26101029e0 0x7f2610198050 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009230 tx=0x7f25f802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:07.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.840+0000 7f260cc00640 1 -- 172.21.15.12:0/2655532029 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f8041070 con 0x7f26101029e0 2024-03-17T23:57:07.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.840+0000 7f2616400640 1 -- 172.21.15.12:0/2655532029 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f261019f9c0 con 0x7f26101029e0 2024-03-17T23:57:07.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.840+0000 7f260cc00640 1 -- 172.21.15.12:0/2655532029 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25f8004450 con 0x7f26101029e0 2024-03-17T23:57:07.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.840+0000 7f260cc00640 1 -- 172.21.15.12:0/2655532029 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f8003db0 con 0x7f26101029e0 2024-03-17T23:57:07.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.840+0000 7f2616400640 1 -- 172.21.15.12:0/2655532029 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f261019feb0 con 0x7f26101029e0 2024-03-17T23:57:07.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.841+0000 7f2616400640 1 -- 172.21.15.12:0/2655532029 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2610108b10 con 0x7f26101029e0 2024-03-17T23:57:07.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.843+0000 7f260cc00640 1 -- 172.21.15.12:0/2655532029 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f25f8044070 con 0x7f26101029e0 2024-03-17T23:57:07.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.844+0000 7f260cc00640 1 --2- 172.21.15.12:0/2655532029 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f25f40a2800 0x7f25f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:07.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.844+0000 7f260cc00640 1 -- 172.21.15.12:0/2655532029 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f25f80f09f0 con 0x7f26101029e0 2024-03-17T23:57:07.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.845+0000 7f260f400640 1 --2- 172.21.15.12:0/2655532029 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f25f40a2800 0x7f25f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:07.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.845+0000 7f260f400640 1 --2- 172.21.15.12:0/2655532029 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f25f40a2800 0x7f25f40a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2610102840 tx=0x7f2600005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:07.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:07.848+0000 7f260cc00640 1 -- 172.21.15.12:0/2655532029 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f25f80bd6b0 con 0x7f26101029e0 2024-03-17T23:57:07.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:07 smithi012 ceph-mon[31541]: pgmap v390: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 490 B/s wr, 12 op/s 2024-03-17T23:57:07.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:07 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2805556287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:07.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:07 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3839273276' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:08.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:07 smithi079 ceph-mon[36241]: pgmap v390: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 490 B/s wr, 12 op/s 2024-03-17T23:57:08.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:07 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2805556287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:08.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:07 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3839273276' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:08.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:07 smithi191 ceph-mon[36708]: pgmap v390: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 490 B/s wr, 12 op/s 2024-03-17T23:57:08.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:07 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2805556287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:08.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:07 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3839273276' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.102+0000 7f2616400640 1 -- 172.21.15.12:0/2655532029 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2610062410 con 0x7f26101029e0 2024-03-17T23:57:08.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.103+0000 7f260cc00640 1 -- 172.21.15.12:0/2655532029 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11936 (secure 0 0 0) 0x7f25f80bcfb0 con 0x7f26101029e0 2024-03-17T23:57:08.103 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:08.104 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-17T23:54:56.312554+0000","modified":"2024-03-17T23:55:28.030481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24575,"mds_2":24530,"mds_3":24529,"mds_4":24514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.12:6841/2073607248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":2073607248},{"type":"v1","addr":"172.21.15.12:6841","nonce":2073607248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/2478742711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":2478742711},{"type":"v1","addr":"172.21.15.79:6839","nonce":2478742711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24575,24587,24530,24529,24514]},"id":1}]} 2024-03-17T23:57:08.104 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 28 2024-03-17T23:57:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.105+0000 7f2616400640 1 -- 172.21.15.12:0/2655532029 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f25f40a2800 msgr2=0x7f25f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.105+0000 7f2616400640 1 --2- 172.21.15.12:0/2655532029 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f25f40a2800 0x7f25f40a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2610102840 tx=0x7f2600005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.105+0000 7f2616400640 1 -- 172.21.15.12:0/2655532029 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26101029e0 msgr2=0x7f2610198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.105+0000 7f2616400640 1 --2- 172.21.15.12:0/2655532029 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26101029e0 0x7f2610198050 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009230 tx=0x7f25f802c650 comp rx=0 tx=0).stop 2024-03-17T23:57:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.105+0000 7f2616400640 1 -- 172.21.15.12:0/2655532029 shutdown_connections 2024-03-17T23:57:08.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.106+0000 7f2616400640 1 --2- 172.21.15.12:0/2655532029 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f25f40a2800 0x7f25f40a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.106+0000 7f2616400640 1 --2- 172.21.15.12:0/2655532029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26101033a0 0x7f261019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.106+0000 7f2616400640 1 --2- 172.21.15.12:0/2655532029 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26101029e0 0x7f2610198050 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.106+0000 7f2616400640 1 --2- 172.21.15.12:0/2655532029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26101017f0 0x7f2610197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.106+0000 7f2616400640 1 -- 172.21.15.12:0/2655532029 >> 172.21.15.12:0/2655532029 conn(0x7f26100fcd80 msgr2=0x7f2610105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.106+0000 7f2616400640 1 -- 172.21.15.12:0/2655532029 shutdown_connections 2024-03-17T23:57:08.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.106+0000 7f2616400640 1 -- 172.21.15.12:0/2655532029 wait complete. 2024-03-17T23:57:08.132 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-17T23:54:56.312554+0000', 'modified': '2024-03-17T23:55:28.030481+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24587, 'mds_1': 24575, 'mds_2': 24530, 'mds_3': 24529, 'mds_4': 24514}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24439': {'gid': 24439, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6833/1787103083', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6832', 'nonce': 1787103083}, {'type': 'v1', 'addr': '172.21.15.191:6833', 'nonce': 1787103083}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.79:6833/843208408', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6832', 'nonce': 843208408}, {'type': 'v1', 'addr': '172.21.15.79:6833', 'nonce': 843208408}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'd', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.12:6837/1104379442', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6836', 'nonce': 1104379442}, {'type': 'v1', 'addr': '172.21.15.12:6837', 'nonce': 1104379442}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24502': {'gid': 24502, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6835/3192827956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 3192827956}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 3192827956}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.79:6835/4099928196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6834', 'nonce': 4099928196}, {'type': 'v1', 'addr': '172.21.15.79:6835', 'nonce': 4099928196}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24514': {'gid': 24514, 'name': 'c', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.191:6837/2977770216', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 2977770216}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 2977770216}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24529': {'gid': 24529, 'name': 'i', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.191:6839/4148729123', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6838', 'nonce': 4148729123}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 4148729123}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.79:6837/3814468390', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6836', 'nonce': 3814468390}, {'type': 'v1', 'addr': '172.21.15.79:6837', 'nonce': 3814468390}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24575': {'gid': 24575, 'name': 'a', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.12:6841/2073607248', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6840', 'nonce': 2073607248}, {'type': 'v1', 'addr': '172.21.15.12:6841', 'nonce': 2073607248}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.79:6839/2478742711', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6838', 'nonce': 2478742711}, {'type': 'v1', 'addr': '172.21.15.79:6839', 'nonce': 2478742711}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24587, 'qdb_cluster': [24575, 24587, 24530, 24529, 24514]} 2024-03-17T23:57:08.133 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-17T23:57:08.133 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-17T23:57:08.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.356+0000 7f6a80400640 1 -- 172.21.15.12:0/3846984106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a780761a0 msgr2=0x7f6a780765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.356+0000 7f6a80400640 1 --2- 172.21.15.12:0/3846984106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a780761a0 0x7f6a780765c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009860 tx=0x7f6a6c02bb70 comp rx=0 tx=0).stop 2024-03-17T23:57:08.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.356+0000 7f6a80400640 1 -- 172.21.15.12:0/3846984106 shutdown_connections 2024-03-17T23:57:08.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.356+0000 7f6a80400640 1 --2- 172.21.15.12:0/3846984106 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a78077d80 0x7f6a78078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.356+0000 7f6a80400640 1 --2- 172.21.15.12:0/3846984106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a780773e0 0x7f6a78077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.356+0000 7f6a80400640 1 --2- 172.21.15.12:0/3846984106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a780761a0 0x7f6a780765c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.356+0000 7f6a80400640 1 -- 172.21.15.12:0/3846984106 >> 172.21.15.12:0/3846984106 conn(0x7f6a780fef90 msgr2=0x7f6a781013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.356+0000 7f6a80400640 1 -- 172.21.15.12:0/3846984106 shutdown_connections 2024-03-17T23:57:08.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.356+0000 7f6a80400640 1 -- 172.21.15.12:0/3846984106 wait complete. 2024-03-17T23:57:08.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.357+0000 7f6a80400640 1 Processor -- start 2024-03-17T23:57:08.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.357+0000 7f6a80400640 1 -- start start 2024-03-17T23:57:08.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.358+0000 7f6a80400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a780761a0 0x7f6a7819e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.358+0000 7f6a80400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a780773e0 0x7f6a7819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.358+0000 7f6a80400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a78077d80 0x7f6a781a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.358+0000 7f6a80400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6a78113a90 con 0x7f6a780761a0 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.358+0000 7f6a80400640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6a78113910 con 0x7f6a78077d80 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.358+0000 7f6a80400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6a78113c10 con 0x7f6a780773e0 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.358+0000 7f6a7da00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a780761a0 0x7f6a7819e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.358+0000 7f6a7d000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a780773e0 0x7f6a7819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.358+0000 7f6a7da00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a780761a0 0x7f6a7819e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46984/0 (socket says 172.21.15.12:46984) 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.358+0000 7f6a7da00640 1 -- 172.21.15.12:0/3969536081 learned_addr learned my addr 172.21.15.12:0/3969536081 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.358+0000 7f6a7d000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a780773e0 0x7f6a7819e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53492/0 (socket says 172.21.15.12:53492) 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.358+0000 7f6a7e400640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a78077d80 0x7f6a781a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.359+0000 7f6a7da00640 1 -- 172.21.15.12:0/3969536081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a780773e0 msgr2=0x7f6a7819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.359+0000 7f6a7da00640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a780773e0 0x7f6a7819e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.359+0000 7f6a7da00640 1 -- 172.21.15.12:0/3969536081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a78077d80 msgr2=0x7f6a781a5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.359+0000 7f6a7da00640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a78077d80 0x7f6a781a5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.359+0000 7f6a7da00640 1 -- 172.21.15.12:0/3969536081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a781a5d90 con 0x7f6a780761a0 2024-03-17T23:57:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.359+0000 7f6a7e400640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a78077d80 0x7f6a781a5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:08.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.359+0000 7f6a7da00640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a780761a0 0x7f6a7819e170 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c037040 tx=0x7f6a6c034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.359+0000 7f6a76000640 1 -- 172.21.15.12:0/3969536081 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c041020 con 0x7f6a780761a0 2024-03-17T23:57:08.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.359+0000 7f6a76000640 1 -- 172.21.15.12:0/3969536081 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a6c02c9f0 con 0x7f6a780761a0 2024-03-17T23:57:08.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.359+0000 7f6a80400640 1 -- 172.21.15.12:0/3969536081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a781a6020 con 0x7f6a780761a0 2024-03-17T23:57:08.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.359+0000 7f6a76000640 1 -- 172.21.15.12:0/3969536081 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c039040 con 0x7f6a780761a0 2024-03-17T23:57:08.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.360+0000 7f6a80400640 1 -- 172.21.15.12:0/3969536081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a781a6560 con 0x7f6a780761a0 2024-03-17T23:57:08.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.363+0000 7f6a83065740 1 Processor -- start 2024-03-17T23:57:08.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.363+0000 7f6a76000640 1 -- 172.21.15.12:0/3969536081 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6a6c044020 con 0x7f6a780761a0 2024-03-17T23:57:08.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.363+0000 7f6a83065740 1 -- start start 2024-03-17T23:57:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.363+0000 7f6a83065740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0a5160 0x55c0eb0a5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.363+0000 7f6a83065740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0aa650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.363+0000 7f6a83065740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0ad150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.363+0000 7f6a83065740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb009c50 con 0x55c0eb0a5160 2024-03-17T23:57:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.363+0000 7f6a83065740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55c0eafb29d0 con 0x55c0eb0aacc0 2024-03-17T23:57:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.363+0000 7f6a83065740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb005cf0 con 0x55c0eb0a5b50 2024-03-17T23:57:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.364+0000 7f6a7da00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0aa650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.364+0000 7f6a7e400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0a5160 0x55c0eb0a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.364+0000 7f6a7d000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.364+0000 7f6a7da00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0aa650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53494/0 (socket says 172.21.15.12:53494) 2024-03-17T23:57:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.364+0000 7f6a7da00640 1 -- 172.21.15.12:0/1995230076 learned_addr learned my addr 172.21.15.12:0/1995230076 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:08.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.364+0000 7f6a7da00640 1 -- 172.21.15.12:0/1995230076 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0aacc0 msgr2=0x55c0eb0ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.364+0000 7f6a7da00640 1 --2- 172.21.15.12:0/1995230076 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0ad150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.364+0000 7f6a7da00640 1 -- 172.21.15.12:0/1995230076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0a5160 msgr2=0x55c0eb0a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.364+0000 7f6a7da00640 1 --2- 172.21.15.12:0/1995230076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0a5160 0x55c0eb0a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.364+0000 7f6a7da00640 1 -- 172.21.15.12:0/1995230076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0eafccf10 con 0x55c0eb0a5b50 2024-03-17T23:57:08.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.364+0000 7f6a7e400640 1 --2- 172.21.15.12:0/1995230076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0a5160 0x55c0eb0a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:08.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.365+0000 7f6a76000640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6a600a2880 0x7f6a600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.365+0000 7f6a7da00640 1 --2- 172.21.15.12:0/1995230076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0aa650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c0f65e0 tx=0x7f6a6c0f66d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20a2a6b5aa53b5bd server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.365+0000 7f6a76000640 1 -- 172.21.15.12:0/3969536081 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6a6c0f0670 con 0x7f6a780761a0 2024-03-17T23:57:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.365+0000 7f6a6aa00640 1 -- 172.21.15.12:0/1995230076 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0b8ed0 con 0x55c0eb0a5b50 2024-03-17T23:57:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.365+0000 7f6a7e400640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6a600a2880 0x7f6a600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.365+0000 7f6a6aa00640 1 -- 172.21.15.12:0/1995230076 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a6c0bd330 con 0x55c0eb0a5b50 2024-03-17T23:57:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.365+0000 7f6a6aa00640 1 -- 172.21.15.12:0/1995230076 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0f6200 con 0x55c0eb0a5b50 2024-03-17T23:57:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.366+0000 7f6a83065740 1 -- 172.21.15.12:0/1995230076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5b50 msgr2=0x55c0eb0aa650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.366+0000 7f6a7e400640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6a600a2880 0x7f6a600a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x55c0eb0a63b0 tx=0x7f6a70010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.366+0000 7f6a83065740 1 --2- 172.21.15.12:0/1995230076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0aa650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c0f65e0 tx=0x7f6a6c0f66d0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.366+0000 7f6a83065740 1 -- 172.21.15.12:0/1995230076 shutdown_connections 2024-03-17T23:57:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.366+0000 7f6a83065740 1 --2- 172.21.15.12:0/1995230076 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0ad150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.366+0000 7f6a83065740 1 --2- 172.21.15.12:0/1995230076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0aa650 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.366+0000 7f6a83065740 1 --2- 172.21.15.12:0/1995230076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0a5160 0x55c0eb0a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.366+0000 7f6a83065740 1 -- 172.21.15.12:0/1995230076 >> 172.21.15.12:0/1995230076 conn(0x55c0eb0a06d0 msgr2=0x55c0eb0a0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.366+0000 7f6a83065740 1 -- 172.21.15.12:0/1995230076 shutdown_connections 2024-03-17T23:57:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.366+0000 7f6a83065740 1 -- 172.21.15.12:0/1995230076 wait complete. 2024-03-17T23:57:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.366+0000 7f6a83065740 1 -- start start 2024-03-17T23:57:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.367+0000 7f6a83065740 1 Processor -- start 2024-03-17T23:57:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.367+0000 7f6a83065740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5160 0x55c0eb0aef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.367+0000 7f6a83065740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.367+0000 7f6a83065740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.367+0000 7f6a83065740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb0ada30 con 0x55c0eb0aacc0 2024-03-17T23:57:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.367+0000 7f6a83065740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55c0eafc5e20 con 0x55c0eb0a5160 2024-03-17T23:57:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.367+0000 7f6a83065740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb0adbb0 con 0x55c0eb0a5b50 2024-03-17T23:57:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.367+0000 7f6a83065740 10 client.0 _ll_register_callbacks cb 0x55c0eafc57f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.367+0000 7f6a7da00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5160 0x55c0eb0aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.367+0000 7f6a7d000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.367+0000 7f6a7da00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5160 0x55c0eb0aef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56780/0 (socket says 172.21.15.12:56780) 2024-03-17T23:57:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.367+0000 7f6a7da00640 1 -- 172.21.15.12:0/3974404098 learned_addr learned my addr 172.21.15.12:0/3974404098 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.367+0000 7f6a7e400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a7da00640 1 -- 172.21.15.12:0/3974404098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5b50 msgr2=0x55c0eb0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a7da00640 1 --2- 172.21.15.12:0/3974404098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a7da00640 1 -- 172.21.15.12:0/3974404098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0aacc0 msgr2=0x55c0eb0b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a7da00640 1 --2- 172.21.15.12:0/3974404098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0b64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a7da00640 1 -- 172.21.15.12:0/3974404098 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0ead1e2d0 con 0x55c0eb0a5160 2024-03-17T23:57:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a7e400640 1 --2- 172.21.15.12:0/3974404098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a7da00640 1 --2- 172.21.15.12:0/3974404098 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5160 0x55c0eb0aef70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c0bc930 tx=0x7f6a6c02bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2675fb727dc6f159 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a5f400640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-03-17T23:57:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a5f400640 1 -- 172.21.15.12:0/3974404098 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0bc590 con 0x55c0eb0a5160 2024-03-17T23:57:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a5f400640 1 -- 172.21.15.12:0/3974404098 <== mon.2 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6a6c004380 con 0x55c0eb0a5160 2024-03-17T23:57:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a83065740 1 -- 172.21.15.12:0/3974404098 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a6c0bd330 con 0x55c0eb0a5160 2024-03-17T23:57:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a5f400640 1 -- 172.21.15.12:0/3974404098 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6a6c0bc770 con 0x55c0eb0a5160 2024-03-17T23:57:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a83065740 10 client.25160 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.368+0000 7f6a83065740 1 -- 172.21.15.12:0/3974404098 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c0eade86b0 con 0x55c0eb0a5160 2024-03-17T23:57:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.369+0000 7f6a5f400640 1 -- 172.21.15.12:0/3974404098 <== mon.2 v2:172.21.15.79:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a6c0bd330 con 0x55c0eb0a5160 2024-03-17T23:57:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.369+0000 7f6a5f400640 1 -- 172.21.15.12:0/3974404098 <== mon.2 v2:172.21.15.79:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6a6c03c030 con 0x55c0eb0a5160 2024-03-17T23:57:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.369+0000 7f6a83065740 10 client.25160 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.369+0000 7f6a83065740 10 client.25160 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-17T23:57:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.369+0000 7f6a83065740 20 client.25160 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.369+0000 7f6a83065740 1 --2- 172.21.15.12:0/3974404098 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x55c0eb0bbae0 0x55c0eb0bcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.369+0000 7f6a83065740 4 client.25160 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:57:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.369+0000 7f6a83065740 1 -- 172.21.15.12:0/3974404098 --> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0eb0672b0 con 0x55c0eb0bbae0 2024-03-17T23:57:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.369+0000 7f6a7d000640 1 --2- 172.21.15.12:0/3974404098 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x55c0eb0bbae0 0x55c0eb0bcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.370+0000 7f6a5f400640 10 client.25160 ms_handle_connect on v2:172.21.15.191:6836/2977770216 2024-03-17T23:57:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.370+0000 7f6a7d000640 1 --2- 172.21.15.12:0/3974404098 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x55c0eb0bbae0 0x55c0eb0bcf50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=89c8e62221d844b4 server_cookie=1d08ea49f477f6d3 in_seq=0 out_seq=0 2024-03-17T23:57:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.371+0000 7f6a5f400640 1 -- 172.21.15.12:0/3974404098 <== mds.4 v2:172.21.15.191:6836/2977770216 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c0eb0672b0 con 0x55c0eb0bbae0 2024-03-17T23:57:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.371+0000 7f6a5f400640 10 client.25160 handle_command_reply: tid=0 2024-03-17T23:57:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.371+0000 7f6a83065740 1 client.25160 shutdown 2024-03-17T23:57:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.371+0000 7f6a83065740 1 -- 172.21.15.12:0/3974404098 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5160 msgr2=0x55c0eb0aef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.371+0000 7f6a83065740 1 --2- 172.21.15.12:0/3974404098 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5160 0x55c0eb0aef70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c0bc930 tx=0x7f6a6c02bb70 comp rx=0 tx=0).stop 2024-03-17T23:57:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.371+0000 7f6a83065740 1 -- 172.21.15.12:0/3974404098 shutdown_connections 2024-03-17T23:57:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.371+0000 7f6a83065740 1 --2- 172.21.15.12:0/3974404098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.371+0000 7f6a83065740 1 --2- 172.21.15.12:0/3974404098 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x55c0eb0bbae0 0x55c0eb0bcf50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.371+0000 7f6a83065740 1 --2- 172.21.15.12:0/3974404098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.371+0000 7f6a83065740 1 --2- 172.21.15.12:0/3974404098 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5160 0x55c0eb0aef70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.371+0000 7f6a5f400640 0 client.25160 ms_handle_reset on v2:172.21.15.191:6836/2977770216 2024-03-17T23:57:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.371+0000 7f6a83065740 1 -- 172.21.15.12:0/3974404098 >> 172.21.15.12:0/3974404098 conn(0x55c0eb0a06d0 msgr2=0x55c0eb09fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.371+0000 7f6a83065740 1 -- 172.21.15.12:0/3974404098 shutdown_connections 2024-03-17T23:57:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.372+0000 7f6a83065740 1 -- 172.21.15.12:0/3974404098 wait complete. 2024-03-17T23:57:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.372+0000 7f6a83065740 20 client.25160 trim_cache size 0 max 16384 2024-03-17T23:57:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.391+0000 7f6a83065740 1 Processor -- start 2024-03-17T23:57:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.391+0000 7f6a83065740 1 -- start start 2024-03-17T23:57:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.392+0000 7f6a83065740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5160 0x55c0eb0af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.392+0000 7f6a83065740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0b6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.392+0000 7f6a83065740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0a65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.392+0000 7f6a83065740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb0bb290 con 0x55c0eb0aacc0 2024-03-17T23:57:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.392+0000 7f6a83065740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb0bb960 con 0x55c0eb0a5b50 2024-03-17T23:57:08.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.392+0000 7f6a83065740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb0bc030 con 0x55c0eb0a5160 2024-03-17T23:57:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.392+0000 7f6a7da00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5160 0x55c0eb0af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.392+0000 7f6a7da00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5160 0x55c0eb0af290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53516/0 (socket says 172.21.15.12:53516) 2024-03-17T23:57:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.392+0000 7f6a7da00640 1 -- 172.21.15.12:0/1960363229 learned_addr learned my addr 172.21.15.12:0/1960363229 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.392+0000 7f6a7d000640 1 --2- 172.21.15.12:0/1960363229 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.392+0000 7f6a7e400640 1 --2- 172.21.15.12:0/1960363229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0a65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.393+0000 7f6a7d000640 1 -- 172.21.15.12:0/1960363229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5160 msgr2=0x55c0eb0af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.393+0000 7f6a7d000640 1 --2- 172.21.15.12:0/1960363229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5160 0x55c0eb0af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.393+0000 7f6a7d000640 1 -- 172.21.15.12:0/1960363229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0aacc0 msgr2=0x55c0eb0a65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.393+0000 7f6a7d000640 1 --2- 172.21.15.12:0/1960363229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0a65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.393+0000 7f6a7d000640 1 -- 172.21.15.12:0/1960363229 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0eb076750 con 0x55c0eb0a5b50 2024-03-17T23:57:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.393+0000 7f6a7da00640 1 --2- 172.21.15.12:0/1960363229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5160 0x55c0eb0af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.393+0000 7f6a7d000640 1 --2- 172.21.15.12:0/1960363229 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0b6480 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x55c0eb0bd8e0 tx=0x7f6a64005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95fc0f8bd6d387dc server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.393+0000 7f6a6a000640 1 -- 172.21.15.12:0/1960363229 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6401f050 con 0x55c0eb0a5b50 2024-03-17T23:57:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.393+0000 7f6a6a000640 1 -- 172.21.15.12:0/1960363229 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a6401f6d0 con 0x55c0eb0a5b50 2024-03-17T23:57:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.393+0000 7f6a6a000640 1 -- 172.21.15.12:0/1960363229 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64010dc0 con 0x55c0eb0a5b50 2024-03-17T23:57:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.394+0000 7f6a83065740 1 -- 172.21.15.12:0/1960363229 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5b50 msgr2=0x55c0eb0b6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.394+0000 7f6a83065740 1 --2- 172.21.15.12:0/1960363229 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0b6480 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x55c0eb0bd8e0 tx=0x7f6a64005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.394+0000 7f6a83065740 1 -- 172.21.15.12:0/1960363229 shutdown_connections 2024-03-17T23:57:08.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.394+0000 7f6a83065740 1 --2- 172.21.15.12:0/1960363229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0a65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.394+0000 7f6a83065740 1 --2- 172.21.15.12:0/1960363229 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0b6480 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.394+0000 7f6a83065740 1 --2- 172.21.15.12:0/1960363229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0a5160 0x55c0eb0af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.394+0000 7f6a83065740 1 -- 172.21.15.12:0/1960363229 >> 172.21.15.12:0/1960363229 conn(0x55c0eb0a06d0 msgr2=0x55c0eb0aeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.394+0000 7f6a83065740 1 -- 172.21.15.12:0/1960363229 shutdown_connections 2024-03-17T23:57:08.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.394+0000 7f6a83065740 1 -- 172.21.15.12:0/1960363229 wait complete. 2024-03-17T23:57:08.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.394+0000 7f6a83065740 1 -- start start 2024-03-17T23:57:08.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.394+0000 7f6a83065740 1 Processor -- start 2024-03-17T23:57:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a83065740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0a5160 0x55c0eb0b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a83065740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a83065740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0b5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a83065740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb0bcdd0 con 0x55c0eb0a5160 2024-03-17T23:57:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a83065740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb0bc700 con 0x55c0eb0a5b50 2024-03-17T23:57:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a83065740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c0eb0bd4a0 con 0x55c0eb0aacc0 2024-03-17T23:57:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a83065740 10 client.0 _ll_register_callbacks cb 0x7f6a6c0f6430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a7d000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a7da00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0a5160 0x55c0eb0b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a7e400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a7e400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0b5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53530/0 (socket says 172.21.15.12:53530) 2024-03-17T23:57:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a7da00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0a5160 0x55c0eb0b22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47022/0 (socket says 172.21.15.12:47022) 2024-03-17T23:57:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a7da00640 1 -- 172.21.15.12:0/868677332 learned_addr learned my addr 172.21.15.12:0/868677332 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.395+0000 7f6a7d000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0b27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56796/0 (socket says 172.21.15.12:56796) 2024-03-17T23:57:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.396+0000 7f6a7e400640 1 -- 172.21.15.12:0/868677332 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5b50 msgr2=0x55c0eb0b27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.396+0000 7f6a7e400640 1 --2- 172.21.15.12:0/868677332 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0b27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.396+0000 7f6a7e400640 1 -- 172.21.15.12:0/868677332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0a5160 msgr2=0x55c0eb0b22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.396+0000 7f6a7e400640 1 --2- 172.21.15.12:0/868677332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0a5160 0x55c0eb0b22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.396+0000 7f6a7e400640 1 -- 172.21.15.12:0/868677332 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0eb0b5df0 con 0x55c0eb0aacc0 2024-03-17T23:57:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.396+0000 7f6a7d000640 1 --2- 172.21.15.12:0/868677332 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.396+0000 7f6a7e400640 1 --2- 172.21.15.12:0/868677332 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0b5800 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6a70003f70 tx=0x7f6a70004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8feb92c9266bb6f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.396+0000 7f6a5f400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-17T23:57:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.396+0000 7f6a5f400640 1 -- 172.21.15.12:0/868677332 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a7002b070 con 0x55c0eb0aacc0 2024-03-17T23:57:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.396+0000 7f6a83065740 1 -- 172.21.15.12:0/868677332 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a6401f6d0 con 0x55c0eb0aacc0 2024-03-17T23:57:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.396+0000 7f6a5f400640 1 -- 172.21.15.12:0/868677332 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6a70004d20 con 0x55c0eb0aacc0 2024-03-17T23:57:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.397+0000 7f6a5f400640 1 -- 172.21.15.12:0/868677332 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6a70036050 con 0x55c0eb0aacc0 2024-03-17T23:57:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.397+0000 7f6a83065740 10 client.25147 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.397+0000 7f6a83065740 1 -- 172.21.15.12:0/868677332 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c0eb122030 con 0x55c0eb0aacc0 2024-03-17T23:57:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.397+0000 7f6a5f400640 1 -- 172.21.15.12:0/868677332 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a6401f6d0 con 0x55c0eb0aacc0 2024-03-17T23:57:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.397+0000 7f6a5f400640 1 -- 172.21.15.12:0/868677332 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6a70026040 con 0x55c0eb0aacc0 2024-03-17T23:57:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.397+0000 7f6a83065740 10 client.25147 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.397+0000 7f6a83065740 10 client.25147 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-17T23:57:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.397+0000 7f6a83065740 20 client.25147 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.397+0000 7f6a83065740 1 --2- 172.21.15.12:0/868677332 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x55c0eb126cb0 0x55c0eb1230f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.397+0000 7f6a83065740 4 client.25147 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:57:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.397+0000 7f6a83065740 1 -- 172.21.15.12:0/868677332 --> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] -- command(tid 0: {"prefix": "status"}) -- 0x55c0eb017190 con 0x55c0eb126cb0 2024-03-17T23:57:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.398+0000 7f6a7da00640 1 --2- 172.21.15.12:0/868677332 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x55c0eb126cb0 0x55c0eb1230f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.398+0000 7f6a7da00640 1 --2- 172.21.15.12:0/868677332 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x55c0eb126cb0 0x55c0eb1230f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8fcddad6687f648e server_cookie=9b2a80cd9a351041 in_seq=0 out_seq=0 2024-03-17T23:57:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.398+0000 7f6a5f400640 10 client.25147 ms_handle_connect on v2:172.21.15.191:6836/2977770216 2024-03-17T23:57:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.398+0000 7f6a5f400640 1 -- 172.21.15.12:0/868677332 <== mds.4 v2:172.21.15.191:6836/2977770216 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55c0eb017190 con 0x55c0eb126cb0 2024-03-17T23:57:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.398+0000 7f6a5f400640 10 client.25147 handle_command_reply: tid=0 2024-03-17T23:57:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.398+0000 7f6a83065740 1 client.25147 shutdown 2024-03-17T23:57:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.398+0000 7f6a83065740 1 -- 172.21.15.12:0/868677332 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0aacc0 msgr2=0x55c0eb0b5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.398+0000 7f6a83065740 1 --2- 172.21.15.12:0/868677332 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0b5800 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6a70003f70 tx=0x7f6a70004ab0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.399+0000 7f6a83065740 1 -- 172.21.15.12:0/868677332 shutdown_connections 2024-03-17T23:57:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.399+0000 7f6a83065740 1 --2- 172.21.15.12:0/868677332 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x55c0eb126cb0 0x55c0eb1230f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.399+0000 7f6a83065740 1 --2- 172.21.15.12:0/868677332 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c0eb0aacc0 0x55c0eb0b5800 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.399+0000 7f6a83065740 1 --2- 172.21.15.12:0/868677332 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c0eb0a5b50 0x55c0eb0b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.399+0000 7f6a83065740 1 --2- 172.21.15.12:0/868677332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0eb0a5160 0x55c0eb0b22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.399+0000 7f6a83065740 1 -- 172.21.15.12:0/868677332 >> 172.21.15.12:0/868677332 conn(0x55c0eb0a06d0 msgr2=0x55c0eb0a3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.399+0000 7f6a5f400640 0 client.25147 ms_handle_reset on v2:172.21.15.191:6836/2977770216 2024-03-17T23:57:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.399+0000 7f6a83065740 1 -- 172.21.15.12:0/868677332 shutdown_connections 2024-03-17T23:57:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.399+0000 7f6a83065740 1 -- 172.21.15.12:0/868677332 wait complete. 2024-03-17T23:57:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.399+0000 7f6a83065740 20 client.25147 trim_cache size 0 max 16384 2024-03-17T23:57:08.406 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-17T23:57:08.406 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:57:08.406 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 4, 2024-03-17T23:57:08.406 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24514, 2024-03-17T23:57:08.406 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-17T23:57:08.406 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-17T23:57:08.406 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-17T23:57:08.406 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 101.37602139800001, 2024-03-17T23:57:08.406 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 28, 2024-03-17T23:57:08.406 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 94, 2024-03-17T23:57:08.406 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 90, 2024-03-17T23:57:08.407 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 155.15898888199999 2024-03-17T23:57:08.407 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-17T23:57:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.400+0000 7f6a80400640 1 -- 172.21.15.12:0/3969536081 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6a600a2880 msgr2=0x7f6a600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.400+0000 7f6a80400640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6a600a2880 0x7f6a600a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x55c0eb0a63b0 tx=0x7f6a70010040 comp rx=0 tx=0).stop 2024-03-17T23:57:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.400+0000 7f6a80400640 1 -- 172.21.15.12:0/3969536081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a780761a0 msgr2=0x7f6a7819e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.400+0000 7f6a80400640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a780761a0 0x7f6a7819e170 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c037040 tx=0x7f6a6c034760 comp rx=0 tx=0).stop 2024-03-17T23:57:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.400+0000 7f6a80400640 1 -- 172.21.15.12:0/3969536081 shutdown_connections 2024-03-17T23:57:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.400+0000 7f6a80400640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6a600a2880 0x7f6a600a4d40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.400+0000 7f6a80400640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a78077d80 0x7f6a781a5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.400+0000 7f6a80400640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a780773e0 0x7f6a7819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.400+0000 7f6a80400640 1 --2- 172.21.15.12:0/3969536081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a780761a0 0x7f6a7819e170 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.400+0000 7f6a80400640 1 -- 172.21.15.12:0/3969536081 >> 172.21.15.12:0/3969536081 conn(0x7f6a780fef90 msgr2=0x7f6a78100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.400+0000 7f6a80400640 1 -- 172.21.15.12:0/3969536081 shutdown_connections 2024-03-17T23:57:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.400+0000 7f6a80400640 1 -- 172.21.15.12:0/3969536081 wait complete. 2024-03-17T23:57:08.420 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-17T23:57:08.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.630+0000 7f3ad8400640 1 -- 172.21.15.12:0/862980075 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ad0104c10 msgr2=0x7f3ad0107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.630+0000 7f3ad8400640 1 --2- 172.21.15.12:0/862980075 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ad0104c10 0x7f3ad0107000 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009860 tx=0x7f3abc02bb70 comp rx=0 tx=0).stop 2024-03-17T23:57:08.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.630+0000 7f3ad8400640 1 -- 172.21.15.12:0/862980075 shutdown_connections 2024-03-17T23:57:08.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.630+0000 7f3ad8400640 1 --2- 172.21.15.12:0/862980075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ad0107540 0x7f3ad0109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.630+0000 7f3ad8400640 1 --2- 172.21.15.12:0/862980075 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ad0104c10 0x7f3ad0107000 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.630+0000 7f3ad8400640 1 --2- 172.21.15.12:0/862980075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ad01022e0 0x7f3ad01046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.630+0000 7f3ad8400640 1 -- 172.21.15.12:0/862980075 >> 172.21.15.12:0/862980075 conn(0x7f3ad00fbc70 msgr2=0x7f3ad00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.630+0000 7f3ad8400640 1 -- 172.21.15.12:0/862980075 shutdown_connections 2024-03-17T23:57:08.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.630+0000 7f3ad8400640 1 -- 172.21.15.12:0/862980075 wait complete. 2024-03-17T23:57:08.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.631+0000 7f3ad8400640 1 Processor -- start 2024-03-17T23:57:08.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.631+0000 7f3ad8400640 1 -- start start 2024-03-17T23:57:08.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad8400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ad01022e0 0x7f3ad0197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad8400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ad0104c10 0x7f3ad01980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad8400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ad0107540 0x7f3ad019f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad8400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad010d450 con 0x7f3ad0104c10 2024-03-17T23:57:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad8400640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad010d2d0 con 0x7f3ad0107540 2024-03-17T23:57:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad8400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad010d5d0 con 0x7f3ad01022e0 2024-03-17T23:57:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad5a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ad01022e0 0x7f3ad0197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad6400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ad0107540 0x7f3ad019f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad5a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ad01022e0 0x7f3ad0197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53538/0 (socket says 172.21.15.12:53538) 2024-03-17T23:57:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad5a00640 1 -- 172.21.15.12:0/2377389591 learned_addr learned my addr 172.21.15.12:0/2377389591 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad5000640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ad0104c10 0x7f3ad01980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad5a00640 1 -- 172.21.15.12:0/2377389591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ad0107540 msgr2=0x7f3ad019f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad5a00640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ad0107540 0x7f3ad019f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad5a00640 1 -- 172.21.15.12:0/2377389591 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ad0104c10 msgr2=0x7f3ad01980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad5a00640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ad0104c10 0x7f3ad01980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad5a00640 1 -- 172.21.15.12:0/2377389591 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad019f7d0 con 0x7f3ad01022e0 2024-03-17T23:57:08.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.632+0000 7f3ad6400640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ad0107540 0x7f3ad019f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:08.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.633+0000 7f3ad5a00640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ad01022e0 0x7f3ad0197bb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e9c0 tx=0x7f3ac400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.633+0000 7f3ace000640 1 -- 172.21.15.12:0/2377389591 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac400cde0 con 0x7f3ad01022e0 2024-03-17T23:57:08.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.633+0000 7f3ace000640 1 -- 172.21.15.12:0/2377389591 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac40035f0 con 0x7f3ad01022e0 2024-03-17T23:57:08.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.633+0000 7f3ace000640 1 -- 172.21.15.12:0/2377389591 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4010690 con 0x7f3ad01022e0 2024-03-17T23:57:08.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.633+0000 7f3ad8400640 1 -- 172.21.15.12:0/2377389591 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad019fac0 con 0x7f3ad01022e0 2024-03-17T23:57:08.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.633+0000 7f3ad8400640 1 -- 172.21.15.12:0/2377389591 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ad01a0030 con 0x7f3ad01022e0 2024-03-17T23:57:08.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.638+0000 7f3adb17b740 1 Processor -- start 2024-03-17T23:57:08.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.638+0000 7f3adb17b740 1 -- start start 2024-03-17T23:57:08.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.638+0000 7f3ace000640 1 -- 172.21.15.12:0/2377389591 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3ac40037a0 con 0x7f3ad01022e0 2024-03-17T23:57:08.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.638+0000 7f3adb17b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 0x55b304f84990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.638+0000 7f3adb17b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84f60 0x55b304f89a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3adb17b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f8a0d0 0x55b304f8c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3adb17b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b304ee4c50 con 0x55b304f8a0d0 2024-03-17T23:57:08.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3adb17b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b304e8d9d0 con 0x55b304f84570 2024-03-17T23:57:08.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3adb17b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b304ee0cf0 con 0x55b304f84f60 2024-03-17T23:57:08.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3ad6400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 0x55b304f84990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3ad5a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84f60 0x55b304f89a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3ad6400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 0x55b304f84990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56810/0 (socket says 172.21.15.12:56810) 2024-03-17T23:57:08.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3ad6400640 1 -- 172.21.15.12:0/1260647563 learned_addr learned my addr 172.21.15.12:0/1260647563 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:08.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3ad5000640 1 --2- 172.21.15.12:0/1260647563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f8a0d0 0x55b304f8c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3ad6400640 1 -- 172.21.15.12:0/1260647563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84f60 msgr2=0x55b304f89a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3ad6400640 1 --2- 172.21.15.12:0/1260647563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84f60 0x55b304f89a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3ad6400640 1 -- 172.21.15.12:0/1260647563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f8a0d0 msgr2=0x55b304f8c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3ad6400640 1 --2- 172.21.15.12:0/1260647563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f8a0d0 0x55b304f8c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3ad6400640 1 -- 172.21.15.12:0/1260647563 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b304ea7f10 con 0x55b304f84570 2024-03-17T23:57:08.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3ad5000640 1 --2- 172.21.15.12:0/1260647563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f8a0d0 0x55b304f8c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:08.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.639+0000 7f3ad6400640 1 --2- 172.21.15.12:0/1260647563 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 0x55b304f84990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3ac80117e0 tx=0x7f3ac8011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a3c21bcbaa688bb server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.640+0000 7f3ac2a00640 1 -- 172.21.15.12:0/1260647563 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8011310 con 0x55b304f84570 2024-03-17T23:57:08.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.640+0000 7f3ac2a00640 1 -- 172.21.15.12:0/1260647563 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac8003e50 con 0x55b304f84570 2024-03-17T23:57:08.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.640+0000 7f3ac2a00640 1 -- 172.21.15.12:0/1260647563 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8004620 con 0x55b304f84570 2024-03-17T23:57:08.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.640+0000 7f3adb17b740 1 -- 172.21.15.12:0/1260647563 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 msgr2=0x55b304f84990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.640+0000 7f3adb17b740 1 --2- 172.21.15.12:0/1260647563 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 0x55b304f84990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3ac80117e0 tx=0x7f3ac8011ca0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.640+0000 7f3adb17b740 1 -- 172.21.15.12:0/1260647563 shutdown_connections 2024-03-17T23:57:08.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.640+0000 7f3adb17b740 1 --2- 172.21.15.12:0/1260647563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f8a0d0 0x55b304f8c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.640+0000 7f3adb17b740 1 --2- 172.21.15.12:0/1260647563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84f60 0x55b304f89a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.640+0000 7f3adb17b740 1 --2- 172.21.15.12:0/1260647563 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 0x55b304f84990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.641+0000 7f3adb17b740 1 -- 172.21.15.12:0/1260647563 >> 172.21.15.12:0/1260647563 conn(0x55b304f7b6d0 msgr2=0x55b304f7bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.641+0000 7f3adb17b740 1 -- 172.21.15.12:0/1260647563 shutdown_connections 2024-03-17T23:57:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.641+0000 7f3adb17b740 1 -- 172.21.15.12:0/1260647563 wait complete. 2024-03-17T23:57:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.641+0000 7f3adb17b740 1 -- start start 2024-03-17T23:57:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.641+0000 7f3ace000640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ab80a2790 0x7f3ab80a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.641+0000 7f3ace000640 1 -- 172.21.15.12:0/2377389591 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f3ac40cf060 con 0x7f3ad01022e0 2024-03-17T23:57:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.641+0000 7f3adb17b740 1 Processor -- start 2024-03-17T23:57:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.641+0000 7f3ad6400640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ab80a2790 0x7f3ab80a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.642+0000 7f3adb17b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84570 0x55b304f8e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.642+0000 7f3adb17b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f84f60 0x55b304f8ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.642+0000 7f3ad6400640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ab80a2790 0x7f3ab80a4c50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8004100 tx=0x7f3ac8009fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.642+0000 7f3adb17b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f8a0d0 0x55b304f95c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.642+0000 7f3adb17b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b304f8d080 con 0x55b304f84f60 2024-03-17T23:57:08.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.642+0000 7f3adb17b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b304ea0e20 con 0x55b304f8a0d0 2024-03-17T23:57:08.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.642+0000 7f3adb17b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b304f8d200 con 0x55b304f84570 2024-03-17T23:57:08.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.642+0000 7f3adb17b740 10 client.0 _ll_register_callbacks cb 0x55b304ea07f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:08.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.642+0000 7f3ad5a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84570 0x55b304f8e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.642+0000 7f3ad5a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84570 0x55b304f8e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53554/0 (socket says 172.21.15.12:53554) 2024-03-17T23:57:08.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.642+0000 7f3ad5a00640 1 -- 172.21.15.12:0/3133061018 learned_addr learned my addr 172.21.15.12:0/3133061018 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:08.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3ad5000640 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f84f60 0x55b304f8ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3ad6400640 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f8a0d0 0x55b304f95c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3ad5a00640 1 -- 172.21.15.12:0/3133061018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f8a0d0 msgr2=0x55b304f95c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3ad5a00640 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f8a0d0 0x55b304f95c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3ad5a00640 1 -- 172.21.15.12:0/3133061018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f84f60 msgr2=0x55b304f8ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3ad5a00640 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f84f60 0x55b304f8ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3ad5a00640 1 -- 172.21.15.12:0/3133061018 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b304bf92d0 con 0x55b304f84570 2024-03-17T23:57:08.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3ad5000640 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f84f60 0x55b304f8ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:08.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3ad6400640 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f8a0d0 0x55b304f95c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:08.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3ad5a00640 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84570 0x55b304f8e680 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400c6a0 tx=0x7f3ac4097320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fad77f41c86d6284 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3ab7400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-17T23:57:08.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3ab7400640 1 -- 172.21.15.12:0/3133061018 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac409afe0 con 0x55b304f84570 2024-03-17T23:57:08.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3ab7400640 1 -- 172.21.15.12:0/3133061018 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f3ac4097cf0 con 0x55b304f84570 2024-03-17T23:57:08.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.643+0000 7f3adb17b740 1 -- 172.21.15.12:0/3133061018 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ac8003e50 con 0x55b304f84570 2024-03-17T23:57:08.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.644+0000 7f3ab7400640 1 -- 172.21.15.12:0/3133061018 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3ac40860b0 con 0x55b304f84570 2024-03-17T23:57:08.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.644+0000 7f3adb17b740 10 client.25159 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:08.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.644+0000 7f3adb17b740 1 -- 172.21.15.12:0/3133061018 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b304cc36b0 con 0x55b304f84570 2024-03-17T23:57:08.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.644+0000 7f3ab7400640 1 -- 172.21.15.12:0/3133061018 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac8003e50 con 0x55b304f84570 2024-03-17T23:57:08.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.644+0000 7f3ab7400640 1 -- 172.21.15.12:0/3133061018 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3ac40d1110 con 0x55b304f84570 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.644+0000 7f3adb17b740 10 client.25159 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.644+0000 7f3adb17b740 10 client.25159 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.644+0000 7f3adb17b740 20 client.25159 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.644+0000 7f3adb17b740 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55b304f9b4a0 0x55b304f9c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.644+0000 7f3adb17b740 4 client.25159 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.644+0000 7f3adb17b740 1 -- 172.21.15.12:0/3133061018 --> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b304f422b0 con 0x55b304f9b4a0 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.645+0000 7f3ad5000640 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55b304f9b4a0 0x55b304f9c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.645+0000 7f3ad5000640 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55b304f9b4a0 0x55b304f9c910 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3d301508caaa71aa server_cookie=fab2d2c1f98e055e in_seq=0 out_seq=0 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.645+0000 7f3ab7400640 10 client.25159 ms_handle_connect on v2:172.21.15.191:6838/4148729123 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3ab7400640 1 -- 172.21.15.12:0/3133061018 <== mds.3 v2:172.21.15.191:6838/4148729123 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b304f422b0 con 0x55b304f9b4a0 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3ab7400640 10 client.25159 handle_command_reply: tid=0 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3adb17b740 1 client.25159 shutdown 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3adb17b740 1 -- 172.21.15.12:0/3133061018 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84570 msgr2=0x55b304f8e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3adb17b740 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84570 0x55b304f8e680 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400c6a0 tx=0x7f3ac4097320 comp rx=0 tx=0).stop 2024-03-17T23:57:08.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3adb17b740 1 -- 172.21.15.12:0/3133061018 shutdown_connections 2024-03-17T23:57:08.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3adb17b740 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55b304f9b4a0 0x55b304f9c910 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3adb17b740 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f8a0d0 0x55b304f95c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3adb17b740 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f84f60 0x55b304f8ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3adb17b740 1 --2- 172.21.15.12:0/3133061018 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84570 0x55b304f8e680 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3ab7400640 0 client.25159 ms_handle_reset on v2:172.21.15.191:6838/4148729123 2024-03-17T23:57:08.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3adb17b740 1 -- 172.21.15.12:0/3133061018 >> 172.21.15.12:0/3133061018 conn(0x55b304f7b6d0 msgr2=0x55b304f7aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3adb17b740 1 -- 172.21.15.12:0/3133061018 shutdown_connections 2024-03-17T23:57:08.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.646+0000 7f3adb17b740 1 -- 172.21.15.12:0/3133061018 wait complete. 2024-03-17T23:57:08.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.647+0000 7f3adb17b740 20 client.25159 trim_cache size 0 max 16384 2024-03-17T23:57:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.669+0000 7f3adb17b740 1 Processor -- start 2024-03-17T23:57:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.669+0000 7f3adb17b740 1 -- start start 2024-03-17T23:57:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.669+0000 7f3adb17b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 0x55b304f8e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3adb17b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84f60 0x55b304f8e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3adb17b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f8a0d0 0x55b304f7a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3adb17b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b304f9ac50 con 0x55b304f8a0d0 2024-03-17T23:57:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3adb17b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b304f9b320 con 0x55b304f84570 2024-03-17T23:57:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3adb17b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b304f9b9f0 con 0x55b304f84f60 2024-03-17T23:57:08.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3ad5a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 0x55b304f8e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3ad6400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f8a0d0 0x55b304f7a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3ad5a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 0x55b304f8e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56826/0 (socket says 172.21.15.12:56826) 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3ad5a00640 1 -- 172.21.15.12:0/3116810864 learned_addr learned my addr 172.21.15.12:0/3116810864 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3ad5000640 1 --2- 172.21.15.12:0/3116810864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84f60 0x55b304f8e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3ad5a00640 1 -- 172.21.15.12:0/3116810864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84f60 msgr2=0x55b304f8e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3ad5a00640 1 --2- 172.21.15.12:0/3116810864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84f60 0x55b304f8e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3ad5a00640 1 -- 172.21.15.12:0/3116810864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f8a0d0 msgr2=0x55b304f7a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3ad5a00640 1 --2- 172.21.15.12:0/3116810864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f8a0d0 0x55b304f7a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3ad5a00640 1 -- 172.21.15.12:0/3116810864 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b304f51750 con 0x55b304f84570 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3ad6400640 1 --2- 172.21.15.12:0/3116810864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f8a0d0 0x55b304f7a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.670+0000 7f3ad5a00640 1 --2- 172.21.15.12:0/3116810864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 0x55b304f8e2b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4099930 tx=0x7f3ac4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf88ef0f6fecdaa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.671+0000 7f3ac2000640 1 -- 172.21.15.12:0/3116810864 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac409afe0 con 0x55b304f84570 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.671+0000 7f3ac2000640 1 -- 172.21.15.12:0/3116810864 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac40bb050 con 0x55b304f84570 2024-03-17T23:57:08.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.671+0000 7f3ac2000640 1 -- 172.21.15.12:0/3116810864 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac409b450 con 0x55b304f84570 2024-03-17T23:57:08.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.671+0000 7f3adb17b740 1 -- 172.21.15.12:0/3116810864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 msgr2=0x55b304f8e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.671+0000 7f3adb17b740 1 --2- 172.21.15.12:0/3116810864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 0x55b304f8e2b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4099930 tx=0x7f3ac4000bc0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.671+0000 7f3adb17b740 1 -- 172.21.15.12:0/3116810864 shutdown_connections 2024-03-17T23:57:08.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.672+0000 7f3adb17b740 1 --2- 172.21.15.12:0/3116810864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f8a0d0 0x55b304f7a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.672+0000 7f3adb17b740 1 --2- 172.21.15.12:0/3116810864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f84f60 0x55b304f8e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.672+0000 7f3adb17b740 1 --2- 172.21.15.12:0/3116810864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84570 0x55b304f8e2b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.672+0000 7f3adb17b740 1 -- 172.21.15.12:0/3116810864 >> 172.21.15.12:0/3116810864 conn(0x55b304f7b6d0 msgr2=0x55b304f7bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.672+0000 7f3adb17b740 1 -- 172.21.15.12:0/3116810864 shutdown_connections 2024-03-17T23:57:08.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.672+0000 7f3adb17b740 1 -- 172.21.15.12:0/3116810864 wait complete. 2024-03-17T23:57:08.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.672+0000 7f3adb17b740 1 -- start start 2024-03-17T23:57:08.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.672+0000 7f3adb17b740 1 Processor -- start 2024-03-17T23:57:08.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.673+0000 7f3adb17b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f84570 0x55b304f82080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.673+0000 7f3adb17b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84f60 0x55b304f825c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.673+0000 7f3adb17b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f8a0d0 0x55b304f95b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.673+0000 7f3adb17b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b304f9c790 con 0x55b304f84570 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.673+0000 7f3adb17b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b304f9c0c0 con 0x55b304f84f60 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.673+0000 7f3adb17b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b304f9ce60 con 0x55b304f8a0d0 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.673+0000 7f3adb17b740 10 client.0 _ll_register_callbacks cb 0x7f3ac4004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.673+0000 7f3ad5000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84f60 0x55b304f825c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.673+0000 7f3ad5000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84f60 0x55b304f825c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56838/0 (socket says 172.21.15.12:56838) 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.673+0000 7f3ad5000640 1 -- 172.21.15.12:0/1674221337 learned_addr learned my addr 172.21.15.12:0/1674221337 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.673+0000 7f3ad6400640 1 --2- 172.21.15.12:0/1674221337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f8a0d0 0x55b304f95b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.674+0000 7f3ad5a00640 1 --2- 172.21.15.12:0/1674221337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f84570 0x55b304f82080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.674+0000 7f3ad5000640 1 -- 172.21.15.12:0/1674221337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f8a0d0 msgr2=0x55b304f95b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.674+0000 7f3ad5000640 1 --2- 172.21.15.12:0/1674221337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f8a0d0 0x55b304f95b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.674+0000 7f3ad5000640 1 -- 172.21.15.12:0/1674221337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f84570 msgr2=0x55b304f82080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.674+0000 7f3ad5000640 1 --2- 172.21.15.12:0/1674221337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f84570 0x55b304f82080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.674+0000 7f3ad5000640 1 -- 172.21.15.12:0/1674221337 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b304f82fc0 con 0x55b304f84f60 2024-03-17T23:57:08.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.674+0000 7f3ad5000640 1 --2- 172.21.15.12:0/1674221337 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84f60 0x55b304f825c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009860 tx=0x7f3abc005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4b66b64b5b2d2d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.674+0000 7f3ab7400640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-03-17T23:57:08.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.674+0000 7f3ab7400640 1 -- 172.21.15.12:0/1674221337 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc04a030 con 0x55b304f84f60 2024-03-17T23:57:08.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.674+0000 7f3adb17b740 1 -- 172.21.15.12:0/1674221337 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ac40bb050 con 0x55b304f84f60 2024-03-17T23:57:08.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.674+0000 7f3ab7400640 1 -- 172.21.15.12:0/1674221337 <== mon.2 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f3abc034500 con 0x55b304f84f60 2024-03-17T23:57:08.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.675+0000 7f3ab7400640 1 -- 172.21.15.12:0/1674221337 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3abc057050 con 0x55b304f84f60 2024-03-17T23:57:08.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.675+0000 7f3adb17b740 10 client.25184 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:08.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.675+0000 7f3adb17b740 1 -- 172.21.15.12:0/1674221337 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b305005a00 con 0x55b304f84f60 2024-03-17T23:57:08.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.675+0000 7f3ab7400640 1 -- 172.21.15.12:0/1674221337 <== mon.2 v2:172.21.15.79:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac40bb050 con 0x55b304f84f60 2024-03-17T23:57:08.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.675+0000 7f3ab7400640 1 -- 172.21.15.12:0/1674221337 <== mon.2 v2:172.21.15.79:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3abc02c510 con 0x55b304f84f60 2024-03-17T23:57:08.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.675+0000 7f3adb17b740 10 client.25184 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:08.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.675+0000 7f3adb17b740 10 client.25184 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-17T23:57:08.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.675+0000 7f3adb17b740 20 client.25184 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:08.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.675+0000 7f3adb17b740 1 --2- 172.21.15.12:0/1674221337 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55b30500a550 0x55b305009260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.675+0000 7f3adb17b740 4 client.25184 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:57:08.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.675+0000 7f3adb17b740 1 -- 172.21.15.12:0/1674221337 --> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] -- command(tid 0: {"prefix": "status"}) -- 0x55b304ef2190 con 0x55b30500a550 2024-03-17T23:57:08.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.676+0000 7f3ad5a00640 1 --2- 172.21.15.12:0/1674221337 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55b30500a550 0x55b305009260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.676+0000 7f3ad5a00640 1 --2- 172.21.15.12:0/1674221337 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55b30500a550 0x55b305009260 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=235dcd40b925fdb8 server_cookie=613d11625b8dd850 in_seq=0 out_seq=0 2024-03-17T23:57:08.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.676+0000 7f3ab7400640 10 client.25184 ms_handle_connect on v2:172.21.15.191:6838/4148729123 2024-03-17T23:57:08.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.676+0000 7f3ab7400640 1 -- 172.21.15.12:0/1674221337 <== mds.3 v2:172.21.15.191:6838/4148729123 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b304ef2190 con 0x55b30500a550 2024-03-17T23:57:08.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.676+0000 7f3ab7400640 10 client.25184 handle_command_reply: tid=0 2024-03-17T23:57:08.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.676+0000 7f3adb17b740 1 client.25184 shutdown 2024-03-17T23:57:08.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.676+0000 7f3adb17b740 1 -- 172.21.15.12:0/1674221337 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84f60 msgr2=0x55b304f825c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.676+0000 7f3adb17b740 1 --2- 172.21.15.12:0/1674221337 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84f60 0x55b304f825c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009860 tx=0x7f3abc005780 comp rx=0 tx=0).stop 2024-03-17T23:57:08.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.677+0000 7f3adb17b740 1 -- 172.21.15.12:0/1674221337 shutdown_connections 2024-03-17T23:57:08.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.677+0000 7f3adb17b740 1 --2- 172.21.15.12:0/1674221337 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55b30500a550 0x55b305009260 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.677+0000 7f3adb17b740 1 --2- 172.21.15.12:0/1674221337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b304f8a0d0 0x55b304f95b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.677+0000 7f3ab7400640 0 client.25184 ms_handle_reset on v2:172.21.15.191:6838/4148729123 2024-03-17T23:57:08.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.677+0000 7f3adb17b740 1 --2- 172.21.15.12:0/1674221337 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b304f84f60 0x55b304f825c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.677+0000 7f3adb17b740 1 --2- 172.21.15.12:0/1674221337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b304f84570 0x55b304f82080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.677+0000 7f3adb17b740 1 -- 172.21.15.12:0/1674221337 >> 172.21.15.12:0/1674221337 conn(0x55b304f7b6d0 msgr2=0x55b304f8e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.677+0000 7f3adb17b740 1 -- 172.21.15.12:0/1674221337 shutdown_connections 2024-03-17T23:57:08.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.677+0000 7f3adb17b740 1 -- 172.21.15.12:0/1674221337 wait complete. 2024-03-17T23:57:08.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.677+0000 7f3adb17b740 20 client.25184 trim_cache size 0 max 16384 2024-03-17T23:57:08.682 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-17T23:57:08.682 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:57:08.682 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 3, 2024-03-17T23:57:08.682 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24529, 2024-03-17T23:57:08.682 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-17T23:57:08.682 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-17T23:57:08.682 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-17T23:57:08.682 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 102.674594488, 2024-03-17T23:57:08.682 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 28, 2024-03-17T23:57:08.683 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 94, 2024-03-17T23:57:08.683 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 90, 2024-03-17T23:57:08.683 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 149.60760056199999 2024-03-17T23:57:08.683 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-17T23:57:08.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.678+0000 7f3ad8400640 1 -- 172.21.15.12:0/2377389591 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ab80a2790 msgr2=0x7f3ab80a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.678+0000 7f3ad8400640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ab80a2790 0x7f3ab80a4c50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8004100 tx=0x7f3ac8009fd0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.678+0000 7f3ad8400640 1 -- 172.21.15.12:0/2377389591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ad01022e0 msgr2=0x7f3ad0197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.678+0000 7f3ad8400640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ad01022e0 0x7f3ad0197bb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e9c0 tx=0x7f3ac400ee80 comp rx=0 tx=0).stop 2024-03-17T23:57:08.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.678+0000 7f3ad8400640 1 -- 172.21.15.12:0/2377389591 shutdown_connections 2024-03-17T23:57:08.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.678+0000 7f3ad8400640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3ab80a2790 0x7f3ab80a4c50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.678+0000 7f3ad8400640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ad0107540 0x7f3ad019f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.678+0000 7f3ad8400640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ad0104c10 0x7f3ad01980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.678+0000 7f3ad8400640 1 --2- 172.21.15.12:0/2377389591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ad01022e0 0x7f3ad0197bb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.678+0000 7f3ad8400640 1 -- 172.21.15.12:0/2377389591 >> 172.21.15.12:0/2377389591 conn(0x7f3ad00fbc70 msgr2=0x7f3ad0104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.678+0000 7f3ad8400640 1 -- 172.21.15.12:0/2377389591 shutdown_connections 2024-03-17T23:57:08.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.678+0000 7f3ad8400640 1 -- 172.21.15.12:0/2377389591 wait complete. 2024-03-17T23:57:08.697 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-17T23:57:08.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.946+0000 7fe7dfe00640 1 -- 172.21.15.12:0/2955456801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7e0104ad0 msgr2=0x7fe7e0104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.946+0000 7fe7dfe00640 1 --2- 172.21.15.12:0/2955456801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7e0104ad0 0x7fe7e0104f50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc009900 tx=0x7fe7cc02bc40 comp rx=0 tx=0).stop 2024-03-17T23:57:08.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.946+0000 7fe7dfe00640 1 -- 172.21.15.12:0/2955456801 shutdown_connections 2024-03-17T23:57:08.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.946+0000 7fe7dfe00640 1 --2- 172.21.15.12:0/2955456801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e0105490 0x7fe7e010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.946+0000 7fe7dfe00640 1 --2- 172.21.15.12:0/2955456801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7e0104ad0 0x7fe7e0104f50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.946+0000 7fe7dfe00640 1 --2- 172.21.15.12:0/2955456801 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7e01038e0 0x7fe7e0103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.946+0000 7fe7dfe00640 1 -- 172.21.15.12:0/2955456801 >> 172.21.15.12:0/2955456801 conn(0x7fe7e00fee50 msgr2=0x7fe7e0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.946+0000 7fe7dfe00640 1 -- 172.21.15.12:0/2955456801 shutdown_connections 2024-03-17T23:57:08.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.946+0000 7fe7dfe00640 1 -- 172.21.15.12:0/2955456801 wait complete. 2024-03-17T23:57:08.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.947+0000 7fe7dfe00640 1 Processor -- start 2024-03-17T23:57:08.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.948+0000 7fe7dfe00640 1 -- start start 2024-03-17T23:57:08.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.948+0000 7fe7dfe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7e01038e0 0x7fe7e0199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.948+0000 7fe7dfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e0104ad0 0x7fe7e019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.948+0000 7fe7dfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7e0105490 0x7fe7e01a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.948+0000 7fe7dfe00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e010f5b0 con 0x7fe7e0104ad0 2024-03-17T23:57:08.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.948+0000 7fe7dfe00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e010f430 con 0x7fe7e01038e0 2024-03-17T23:57:08.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.948+0000 7fe7dfe00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e010f730 con 0x7fe7e0105490 2024-03-17T23:57:08.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.948+0000 7fe7dea00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7e01038e0 0x7fe7e0199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.948+0000 7fe7dea00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7e01038e0 0x7fe7e0199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56850/0 (socket says 172.21.15.12:56850) 2024-03-17T23:57:08.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.948+0000 7fe7dea00640 1 -- 172.21.15.12:0/2332792661 learned_addr learned my addr 172.21.15.12:0/2332792661 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:08.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.948+0000 7fe7de000640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e0104ad0 0x7fe7e019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.948+0000 7fe7df400640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7e0105490 0x7fe7e01a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.949+0000 7fe7dea00640 1 -- 172.21.15.12:0/2332792661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7e0105490 msgr2=0x7fe7e01a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.949+0000 7fe7dea00640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7e0105490 0x7fe7e01a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.949+0000 7fe7dea00640 1 -- 172.21.15.12:0/2332792661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e0104ad0 msgr2=0x7fe7e019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.949+0000 7fe7dea00640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e0104ad0 0x7fe7e019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.949+0000 7fe7dea00640 1 -- 172.21.15.12:0/2332792661 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e01a1810 con 0x7fe7e01038e0 2024-03-17T23:57:08.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.949+0000 7fe7dea00640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7e01038e0 0x7fe7e0199bf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe7d000e9c0 tx=0x7fe7d000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.949+0000 7fe7d7400640 1 -- 172.21.15.12:0/2332792661 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d000cd80 con 0x7fe7e01038e0 2024-03-17T23:57:08.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.949+0000 7fe7d7400640 1 -- 172.21.15.12:0/2332792661 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7d00035f0 con 0x7fe7e01038e0 2024-03-17T23:57:08.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.949+0000 7fe7d7400640 1 -- 172.21.15.12:0/2332792661 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0010690 con 0x7fe7e01038e0 2024-03-17T23:57:08.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.949+0000 7fe7dfe00640 1 -- 172.21.15.12:0/2332792661 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e01a1b00 con 0x7fe7e01038e0 2024-03-17T23:57:08.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.950+0000 7fe7dfe00640 1 -- 172.21.15.12:0/2332792661 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e01a1fd0 con 0x7fe7e01038e0 2024-03-17T23:57:08.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.954+0000 7fe7e89f9740 1 Processor -- start 2024-03-17T23:57:08.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.954+0000 7fe7e89f9740 1 -- start start 2024-03-17T23:57:08.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.954+0000 7fe7d7400640 1 -- 172.21.15.12:0/2332792661 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe7d00037a0 con 0x7fe7e01038e0 2024-03-17T23:57:08.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.954+0000 7fe7e89f9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5ee340 0x5620cf5ee760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.954+0000 7fe7e89f9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 0x5620cf5f3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.954+0000 7fe7e89f9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 0x5620cf5f6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.954+0000 7fe7e89f9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5620cf552c50 con 0x5620cf5ee340 2024-03-17T23:57:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.954+0000 7fe7e89f9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5620cf4fb9d0 con 0x5620cf5f3ea0 2024-03-17T23:57:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.954+0000 7fe7e89f9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5620cf54ecf0 con 0x5620cf5eed30 2024-03-17T23:57:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.955+0000 7fe7dea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 0x5620cf5f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.955+0000 7fe7dea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 0x5620cf5f3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53598/0 (socket says 172.21.15.12:53598) 2024-03-17T23:57:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.955+0000 7fe7dea00640 1 -- 172.21.15.12:0/2345929551 learned_addr learned my addr 172.21.15.12:0/2345929551 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.955+0000 7fe7df400640 1 --2- 172.21.15.12:0/2345929551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5ee340 0x5620cf5ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.955+0000 7fe7de000640 1 --2- 172.21.15.12:0/2345929551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 0x5620cf5f6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.955+0000 7fe7dea00640 1 -- 172.21.15.12:0/2345929551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 msgr2=0x5620cf5f6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.955+0000 7fe7dea00640 1 --2- 172.21.15.12:0/2345929551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 0x5620cf5f6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.955+0000 7fe7dea00640 1 -- 172.21.15.12:0/2345929551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5ee340 msgr2=0x5620cf5ee760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.955+0000 7fe7dea00640 1 --2- 172.21.15.12:0/2345929551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5ee340 0x5620cf5ee760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.955+0000 7fe7dea00640 1 -- 172.21.15.12:0/2345929551 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620cf515f10 con 0x5620cf5eed30 2024-03-17T23:57:08.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.955+0000 7fe7de000640 1 --2- 172.21.15.12:0/2345929551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 0x5620cf5f6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.955+0000 7fe7dea00640 1 --2- 172.21.15.12:0/2345929551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 0x5620cf5f3830 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00d7110 tx=0x7fe7d00d7200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68850651c8b13b83 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.956+0000 7fe7cbe00640 1 -- 172.21.15.12:0/2345929551 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d00e2070 con 0x5620cf5eed30 2024-03-17T23:57:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.956+0000 7fe7cbe00640 1 -- 172.21.15.12:0/2345929551 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7d00cf500 con 0x5620cf5eed30 2024-03-17T23:57:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.956+0000 7fe7cbe00640 1 -- 172.21.15.12:0/2345929551 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d00dd640 con 0x5620cf5eed30 2024-03-17T23:57:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.956+0000 7fe7d7400640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe7c40a2880 0x7fe7c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.956+0000 7fe7d7400640 1 -- 172.21.15.12:0/2332792661 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fe7d00cf110 con 0x7fe7e01038e0 2024-03-17T23:57:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.956+0000 7fe7df400640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe7c40a2880 0x7fe7c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.957+0000 7fe7e89f9740 1 -- 172.21.15.12:0/2345929551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 msgr2=0x5620cf5f3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.957+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/2345929551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 0x5620cf5f3830 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00d7110 tx=0x7fe7d00d7200 comp rx=0 tx=0).stop 2024-03-17T23:57:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.957+0000 7fe7e89f9740 1 -- 172.21.15.12:0/2345929551 shutdown_connections 2024-03-17T23:57:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.957+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/2345929551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 0x5620cf5f6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.957+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/2345929551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 0x5620cf5f3830 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.957+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/2345929551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5ee340 0x5620cf5ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.957+0000 7fe7e89f9740 1 -- 172.21.15.12:0/2345929551 >> 172.21.15.12:0/2345929551 conn(0x5620cf5e98b0 msgr2=0x5620cf5e9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.957+0000 7fe7e89f9740 1 -- 172.21.15.12:0/2345929551 shutdown_connections 2024-03-17T23:57:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.957+0000 7fe7df400640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe7c40a2880 0x7fe7c40a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x5620cf5ef590 tx=0x7fe7d8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.957+0000 7fe7e89f9740 1 -- 172.21.15.12:0/2345929551 wait complete. 2024-03-17T23:57:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.958+0000 7fe7e89f9740 1 -- start start 2024-03-17T23:57:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.958+0000 7fe7e89f9740 1 Processor -- start 2024-03-17T23:57:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.958+0000 7fe7e89f9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5ee340 0x5620cf5f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.958+0000 7fe7e89f9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5eed30 0x5620cf5f8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.958+0000 7fe7e89f9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 0x5620cf5ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.958+0000 7fe7e89f9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5620cf5f6e50 con 0x5620cf5eed30 2024-03-17T23:57:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.958+0000 7fe7e89f9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5620cf50ee20 con 0x5620cf5f3ea0 2024-03-17T23:57:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.958+0000 7fe7e89f9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5620cf5f6fd0 con 0x5620cf5ee340 2024-03-17T23:57:08.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.958+0000 7fe7e89f9740 10 client.0 _ll_register_callbacks cb 0x5620cf50e7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.958+0000 7fe7dea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5ee340 0x5620cf5f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7df400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 0x5620cf5ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7dea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5ee340 0x5620cf5f84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53608/0 (socket says 172.21.15.12:53608) 2024-03-17T23:57:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7dea00640 1 -- 172.21.15.12:0/1447729137 learned_addr learned my addr 172.21.15.12:0/1447729137 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7de000640 1 --2- 172.21.15.12:0/1447729137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5eed30 0x5620cf5f8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7dea00640 1 -- 172.21.15.12:0/1447729137 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 msgr2=0x5620cf5ffa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7dea00640 1 --2- 172.21.15.12:0/1447729137 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 0x5620cf5ffa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7dea00640 1 -- 172.21.15.12:0/1447729137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5eed30 msgr2=0x5620cf5f8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:08.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7dea00640 1 --2- 172.21.15.12:0/1447729137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5eed30 0x5620cf5f8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7dea00640 1 -- 172.21.15.12:0/1447729137 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620cf2672d0 con 0x5620cf5ee340 2024-03-17T23:57:08.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7de000640 1 --2- 172.21.15.12:0/1447729137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5eed30 0x5620cf5f8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7dea00640 1 --2- 172.21.15.12:0/1447729137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5ee340 0x5620cf5f84c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00d70e0 tx=0x7fe7d0003e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc2b2bddbbf0ba80 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:08.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-17T23:57:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7c8c00640 1 -- 172.21.15.12:0/1447729137 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0097970 con 0x5620cf5ee340 2024-03-17T23:57:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7c8c00640 1 -- 172.21.15.12:0/1447729137 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fe7d00d12e0 con 0x5620cf5ee340 2024-03-17T23:57:08.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.959+0000 7fe7e89f9740 1 -- 172.21.15.12:0/1447729137 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7d00cf500 con 0x5620cf5ee340 2024-03-17T23:57:08.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.960+0000 7fe7c8c00640 1 -- 172.21.15.12:0/1447729137 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe7d00d2c40 con 0x5620cf5ee340 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.960+0000 7fe7e89f9740 10 client.25177 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.960+0000 7fe7e89f9740 1 -- 172.21.15.12:0/1447729137 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5620cf3316b0 con 0x5620cf5ee340 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.960+0000 7fe7c8c00640 1 -- 172.21.15.12:0/1447729137 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7d00cf500 con 0x5620cf5ee340 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.960+0000 7fe7c8c00640 1 -- 172.21.15.12:0/1447729137 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe7d009b930 con 0x5620cf5ee340 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.960+0000 7fe7e89f9740 10 client.25177 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.960+0000 7fe7e89f9740 10 client.25177 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.960+0000 7fe7e89f9740 20 client.25177 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.960+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/1447729137 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x5620cf603f30 0x5620cf606040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.960+0000 7fe7e89f9740 4 client.25177 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.960+0000 7fe7e89f9740 1 -- 172.21.15.12:0/1447729137 --> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620cf5b02b0 con 0x5620cf603f30 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.961+0000 7fe7de000640 1 --2- 172.21.15.12:0/1447729137 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x5620cf603f30 0x5620cf606040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.961+0000 7fe7de000640 1 --2- 172.21.15.12:0/1447729137 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x5620cf603f30 0x5620cf606040 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=45e120f8e6f3ff9b server_cookie=8a41bd109ac271a1 in_seq=0 out_seq=0 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.961+0000 7fe7c8c00640 10 client.25177 ms_handle_connect on v2:172.21.15.79:6836/3814468390 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.962+0000 7fe7c8c00640 1 -- 172.21.15.12:0/1447729137 <== mds.2 v2:172.21.15.79:6836/3814468390 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5620cf5b02b0 con 0x5620cf603f30 2024-03-17T23:57:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.962+0000 7fe7c8c00640 10 client.25177 handle_command_reply: tid=0 2024-03-17T23:57:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.962+0000 7fe7e89f9740 1 client.25177 shutdown 2024-03-17T23:57:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.962+0000 7fe7e89f9740 1 -- 172.21.15.12:0/1447729137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5ee340 msgr2=0x5620cf5f84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.962+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/1447729137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5ee340 0x5620cf5f84c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00d70e0 tx=0x7fe7d0003e10 comp rx=0 tx=0).stop 2024-03-17T23:57:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.962+0000 7fe7e89f9740 1 -- 172.21.15.12:0/1447729137 shutdown_connections 2024-03-17T23:57:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.962+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/1447729137 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x5620cf603f30 0x5620cf606040 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.962+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/1447729137 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 0x5620cf5ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.962+0000 7fe7c8c00640 0 client.25177 ms_handle_reset on v2:172.21.15.79:6836/3814468390 2024-03-17T23:57:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.962+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/1447729137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5eed30 0x5620cf5f8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.962+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/1447729137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5ee340 0x5620cf5f84c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.962+0000 7fe7e89f9740 1 -- 172.21.15.12:0/1447729137 >> 172.21.15.12:0/1447729137 conn(0x5620cf5e98b0 msgr2=0x5620cf5e8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.962+0000 7fe7e89f9740 1 -- 172.21.15.12:0/1447729137 shutdown_connections 2024-03-17T23:57:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.963+0000 7fe7e89f9740 1 -- 172.21.15.12:0/1447729137 wait complete. 2024-03-17T23:57:08.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.963+0000 7fe7e89f9740 20 client.25177 trim_cache size 0 max 16384 2024-03-17T23:57:08.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.998+0000 7fe7e89f9740 1 Processor -- start 2024-03-17T23:57:08.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.999+0000 7fe7e89f9740 1 -- start start 2024-03-17T23:57:08.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.999+0000 7fe7e89f9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5ee340 0x5620cf5f87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.999+0000 7fe7e89f9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5eed30 0x5620cf5ff9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:08.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.999+0000 7fe7e89f9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5f3ea0 0x5620cf5e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.999+0000 7fe7e89f9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5620cf604380 con 0x5620cf5f3ea0 2024-03-17T23:57:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.999+0000 7fe7e89f9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5620cf604a50 con 0x5620cf5eed30 2024-03-17T23:57:09.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.999+0000 7fe7e89f9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5620cf605120 con 0x5620cf5ee340 2024-03-17T23:57:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.999+0000 7fe7df400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5f3ea0 0x5620cf5e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.999+0000 7fe7de000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5eed30 0x5620cf5ff9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.999+0000 7fe7de000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5eed30 0x5620cf5ff9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56872/0 (socket says 172.21.15.12:56872) 2024-03-17T23:57:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.999+0000 7fe7de000640 1 -- 172.21.15.12:0/1382958938 learned_addr learned my addr 172.21.15.12:0/1382958938 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:08.999+0000 7fe7dea00640 1 --2- 172.21.15.12:0/1382958938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5ee340 0x5620cf5f87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.000+0000 7fe7de000640 1 -- 172.21.15.12:0/1382958938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5ee340 msgr2=0x5620cf5f87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.000+0000 7fe7de000640 1 --2- 172.21.15.12:0/1382958938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5ee340 0x5620cf5f87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.000+0000 7fe7de000640 1 -- 172.21.15.12:0/1382958938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5f3ea0 msgr2=0x5620cf5e81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.000+0000 7fe7de000640 1 --2- 172.21.15.12:0/1382958938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5f3ea0 0x5620cf5e81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.000+0000 7fe7de000640 1 -- 172.21.15.12:0/1382958938 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620cf5f3100 con 0x5620cf5eed30 2024-03-17T23:57:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.000+0000 7fe7dea00640 1 --2- 172.21.15.12:0/1382958938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5ee340 0x5620cf5f87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.000+0000 7fe7de000640 1 --2- 172.21.15.12:0/1382958938 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5eed30 0x5620cf5ff9d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc005ab0 tx=0x7fe7cc009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c92311e2964d71c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.000+0000 7fe7cb400640 1 -- 172.21.15.12:0/1382958938 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc048aa0 con 0x5620cf5eed30 2024-03-17T23:57:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.000+0000 7fe7cb400640 1 -- 172.21.15.12:0/1382958938 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7cc04c030 con 0x5620cf5eed30 2024-03-17T23:57:09.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.000+0000 7fe7cb400640 1 -- 172.21.15.12:0/1382958938 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc0046c0 con 0x5620cf5eed30 2024-03-17T23:57:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.001+0000 7fe7e89f9740 1 -- 172.21.15.12:0/1382958938 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5eed30 msgr2=0x5620cf5ff9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.001+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/1382958938 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5eed30 0x5620cf5ff9d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc005ab0 tx=0x7fe7cc009260 comp rx=0 tx=0).stop 2024-03-17T23:57:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.001+0000 7fe7e89f9740 1 -- 172.21.15.12:0/1382958938 shutdown_connections 2024-03-17T23:57:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.001+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/1382958938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5f3ea0 0x5620cf5e81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.001+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/1382958938 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5eed30 0x5620cf5ff9d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.001+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/1382958938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5ee340 0x5620cf5f87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.001+0000 7fe7e89f9740 1 -- 172.21.15.12:0/1382958938 >> 172.21.15.12:0/1382958938 conn(0x5620cf5e98b0 msgr2=0x5620cf5f80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.001+0000 7fe7e89f9740 1 -- 172.21.15.12:0/1382958938 shutdown_connections 2024-03-17T23:57:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.001+0000 7fe7e89f9740 1 -- 172.21.15.12:0/1382958938 wait complete. 2024-03-17T23:57:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.002+0000 7fe7e89f9740 1 -- start start 2024-03-17T23:57:09.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.002+0000 7fe7e89f9740 1 Processor -- start 2024-03-17T23:57:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.002+0000 7fe7e89f9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5ee340 0x5620cf5fbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.002+0000 7fe7e89f9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 0x5620cf5fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.002+0000 7fe7e89f9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 0x5620cf5fea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.002+0000 7fe7e89f9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5620cf605ec0 con 0x5620cf5ee340 2024-03-17T23:57:09.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.002+0000 7fe7e89f9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5620cf6057f0 con 0x5620cf5f3ea0 2024-03-17T23:57:09.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.002+0000 7fe7e89f9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5620cf606590 con 0x5620cf5eed30 2024-03-17T23:57:09.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.002+0000 7fe7e89f9740 10 client.0 _ll_register_callbacks cb 0x7fe7d00d2e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:09.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.003+0000 7fe7de000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 0x5620cf5fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.003+0000 7fe7dea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5ee340 0x5620cf5fbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.003+0000 7fe7df400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 0x5620cf5fea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.003+0000 7fe7de000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 0x5620cf5fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53636/0 (socket says 172.21.15.12:53636) 2024-03-17T23:57:09.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.003+0000 7fe7de000640 1 -- 172.21.15.12:0/3475342449 learned_addr learned my addr 172.21.15.12:0/3475342449 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:09.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.003+0000 7fe7de000640 1 -- 172.21.15.12:0/3475342449 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 msgr2=0x5620cf5fea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.003+0000 7fe7de000640 1 --2- 172.21.15.12:0/3475342449 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 0x5620cf5fea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.003+0000 7fe7de000640 1 -- 172.21.15.12:0/3475342449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5ee340 msgr2=0x5620cf5fbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.003+0000 7fe7de000640 1 --2- 172.21.15.12:0/3475342449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5ee340 0x5620cf5fbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.003+0000 7fe7de000640 1 -- 172.21.15.12:0/3475342449 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620cf5ff080 con 0x5620cf5eed30 2024-03-17T23:57:09.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.003+0000 7fe7de000640 1 --2- 172.21.15.12:0/3475342449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 0x5620cf5fc0d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc036040 tx=0x7fe7cc02c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a1c9134ca90852b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.003+0000 7fe7c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-17T23:57:09.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.004+0000 7fe7c8c00640 1 -- 172.21.15.12:0/3475342449 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc004230 con 0x5620cf5eed30 2024-03-17T23:57:09.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.004+0000 7fe7c8c00640 1 -- 172.21.15.12:0/3475342449 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fe7cc0043d0 con 0x5620cf5eed30 2024-03-17T23:57:09.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.004+0000 7fe7e89f9740 1 -- 172.21.15.12:0/3475342449 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7cc04c030 con 0x5620cf5eed30 2024-03-17T23:57:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.004+0000 7fe7c8c00640 1 -- 172.21.15.12:0/3475342449 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe7cc060050 con 0x5620cf5eed30 2024-03-17T23:57:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.004+0000 7fe7e89f9740 10 client.25183 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.004+0000 7fe7e89f9740 1 -- 172.21.15.12:0/3475342449 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5620cf673510 con 0x5620cf5eed30 2024-03-17T23:57:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.004+0000 7fe7c8c00640 1 -- 172.21.15.12:0/3475342449 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7cc04c030 con 0x5620cf5eed30 2024-03-17T23:57:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.004+0000 7fe7c8c00640 1 -- 172.21.15.12:0/3475342449 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe7cc005420 con 0x5620cf5eed30 2024-03-17T23:57:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.004+0000 7fe7e89f9740 10 client.25183 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.004+0000 7fe7e89f9740 10 client.25183 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-17T23:57:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.004+0000 7fe7e89f9740 20 client.25183 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.005+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/3475342449 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x5620cf677c70 0x5620cf676980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.005+0000 7fe7e89f9740 4 client.25183 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:57:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.005+0000 7fe7e89f9740 1 -- 172.21.15.12:0/3475342449 --> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] -- command(tid 0: {"prefix": "status"}) -- 0x5620cf560190 con 0x5620cf677c70 2024-03-17T23:57:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.005+0000 7fe7dea00640 1 --2- 172.21.15.12:0/3475342449 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x5620cf677c70 0x5620cf676980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.005+0000 7fe7dea00640 1 --2- 172.21.15.12:0/3475342449 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x5620cf677c70 0x5620cf676980 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4c6376247a362901 server_cookie=ee86686bba4cf860 in_seq=0 out_seq=0 2024-03-17T23:57:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.005+0000 7fe7c8c00640 10 client.25183 ms_handle_connect on v2:172.21.15.79:6836/3814468390 2024-03-17T23:57:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.005+0000 7fe7c8c00640 1 -- 172.21.15.12:0/3475342449 <== mds.2 v2:172.21.15.79:6836/3814468390 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5620cf560190 con 0x5620cf677c70 2024-03-17T23:57:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.005+0000 7fe7c8c00640 10 client.25183 handle_command_reply: tid=0 2024-03-17T23:57:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.005+0000 7fe7e89f9740 1 client.25183 shutdown 2024-03-17T23:57:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.006+0000 7fe7e89f9740 1 -- 172.21.15.12:0/3475342449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 msgr2=0x5620cf5fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.006+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/3475342449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 0x5620cf5fc0d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc036040 tx=0x7fe7cc02c2e0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.006+0000 7fe7e89f9740 1 -- 172.21.15.12:0/3475342449 shutdown_connections 2024-03-17T23:57:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.006+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/3475342449 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x5620cf677c70 0x5620cf676980 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.006+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/3475342449 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5620cf5f3ea0 0x5620cf5fea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.006+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/3475342449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5620cf5eed30 0x5620cf5fc0d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.006+0000 7fe7e89f9740 1 --2- 172.21.15.12:0/3475342449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5620cf5ee340 0x5620cf5fbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.006+0000 7fe7e89f9740 1 -- 172.21.15.12:0/3475342449 >> 172.21.15.12:0/3475342449 conn(0x5620cf5e98b0 msgr2=0x5620cf5e81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.006+0000 7fe7c8c00640 0 client.25183 ms_handle_reset on v2:172.21.15.79:6836/3814468390 2024-03-17T23:57:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.006+0000 7fe7e89f9740 1 -- 172.21.15.12:0/3475342449 shutdown_connections 2024-03-17T23:57:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.006+0000 7fe7e89f9740 1 -- 172.21.15.12:0/3475342449 wait complete. 2024-03-17T23:57:09.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.006+0000 7fe7e89f9740 20 client.25183 trim_cache size 0 max 16384 2024-03-17T23:57:09.013 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-17T23:57:09.014 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:57:09.014 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 2, 2024-03-17T23:57:09.014 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24530, 2024-03-17T23:57:09.014 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-17T23:57:09.014 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-17T23:57:09.014 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-17T23:57:09.014 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 104.019030673, 2024-03-17T23:57:09.014 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 28, 2024-03-17T23:57:09.014 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 94, 2024-03-17T23:57:09.014 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 90, 2024-03-17T23:57:09.014 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 152.88396663399999 2024-03-17T23:57:09.014 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-17T23:57:09.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.006+0000 7fe7dfe00640 1 -- 172.21.15.12:0/2332792661 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe7c40a2880 msgr2=0x7fe7c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.007+0000 7fe7dfe00640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe7c40a2880 0x7fe7c40a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x5620cf5ef590 tx=0x7fe7d8009500 comp rx=0 tx=0).stop 2024-03-17T23:57:09.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.007+0000 7fe7dfe00640 1 -- 172.21.15.12:0/2332792661 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7e01038e0 msgr2=0x7fe7e0199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.007+0000 7fe7dfe00640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7e01038e0 0x7fe7e0199bf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe7d000e9c0 tx=0x7fe7d000ee80 comp rx=0 tx=0).stop 2024-03-17T23:57:09.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.007+0000 7fe7dfe00640 1 -- 172.21.15.12:0/2332792661 shutdown_connections 2024-03-17T23:57:09.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.007+0000 7fe7dfe00640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe7c40a2880 0x7fe7c40a4d40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.007+0000 7fe7dfe00640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7e0105490 0x7fe7e01a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.007+0000 7fe7dfe00640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e0104ad0 0x7fe7e019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.007+0000 7fe7dfe00640 1 --2- 172.21.15.12:0/2332792661 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7e01038e0 0x7fe7e0199bf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.007+0000 7fe7dfe00640 1 -- 172.21.15.12:0/2332792661 >> 172.21.15.12:0/2332792661 conn(0x7fe7e00fee50 msgr2=0x7fe7e0107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.007+0000 7fe7dfe00640 1 -- 172.21.15.12:0/2332792661 shutdown_connections 2024-03-17T23:57:09.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.007+0000 7fe7dfe00640 1 -- 172.21.15.12:0/2332792661 wait complete. 2024-03-17T23:57:09.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:08 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2655532029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:09.027 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-17T23:57:09.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:08 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2655532029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:09.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:08 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2655532029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:09.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.236+0000 7f8332c00640 1 -- 172.21.15.12:0/2131920166 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f832c1016e0 msgr2=0x7f832c101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.236+0000 7f8332c00640 1 --2- 172.21.15.12:0/2131920166 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f832c1016e0 0x7f832c101b00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8320009ca0 tx=0x7f832002c0f0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.236+0000 7f8332c00640 1 -- 172.21.15.12:0/2131920166 shutdown_connections 2024-03-17T23:57:09.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.236+0000 7f8332c00640 1 --2- 172.21.15.12:0/2131920166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f832c103290 0x7f832c109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.236+0000 7f8332c00640 1 --2- 172.21.15.12:0/2131920166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f832c1028d0 0x7f832c102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.236+0000 7f8332c00640 1 --2- 172.21.15.12:0/2131920166 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f832c1016e0 0x7f832c101b00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.236+0000 7f8332c00640 1 -- 172.21.15.12:0/2131920166 >> 172.21.15.12:0/2131920166 conn(0x7f832c0fcc50 msgr2=0x7f832c0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.236+0000 7f8332c00640 1 -- 172.21.15.12:0/2131920166 shutdown_connections 2024-03-17T23:57:09.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.236+0000 7f8332c00640 1 -- 172.21.15.12:0/2131920166 wait complete. 2024-03-17T23:57:09.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.237+0000 7f8332c00640 1 Processor -- start 2024-03-17T23:57:09.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.238+0000 7f8332c00640 1 -- start start 2024-03-17T23:57:09.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.238+0000 7f8332c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f832c1016e0 0x7f832c075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.238+0000 7f8332c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f832c1028d0 0x7f832c073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.238+0000 7f8332c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f832c103290 0x7f832c074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.238+0000 7f8332c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f832c10d360 con 0x7f832c103290 2024-03-17T23:57:09.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.238+0000 7f8332c00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f832c10d1e0 con 0x7f832c1028d0 2024-03-17T23:57:09.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.238+0000 7f8332c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f832c10d4e0 con 0x7f832c1016e0 2024-03-17T23:57:09.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.238+0000 7f8331800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f832c1016e0 0x7f832c075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.239+0000 7f8331800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f832c1016e0 0x7f832c075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53662/0 (socket says 172.21.15.12:53662) 2024-03-17T23:57:09.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.239+0000 7f8331800640 1 -- 172.21.15.12:0/2986906645 learned_addr learned my addr 172.21.15.12:0/2986906645 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:09.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.239+0000 7f8330e00640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f832c1028d0 0x7f832c073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.239+0000 7f8332200640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f832c103290 0x7f832c074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.239+0000 7f8331800640 1 -- 172.21.15.12:0/2986906645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f832c1028d0 msgr2=0x7f832c073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.239+0000 7f8331800640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f832c1028d0 0x7f832c073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.239+0000 7f8331800640 1 -- 172.21.15.12:0/2986906645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f832c103290 msgr2=0x7f832c074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.239+0000 7f8331800640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f832c103290 0x7f832c074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.239+0000 7f8331800640 1 -- 172.21.15.12:0/2986906645 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832c0749a0 con 0x7f832c1016e0 2024-03-17T23:57:09.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.239+0000 7f8332200640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f832c103290 0x7f832c074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:09.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.239+0000 7f8331800640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f832c1016e0 0x7f832c075590 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8320000f80 tx=0x7f8320032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.239+0000 7f832a000640 1 -- 172.21.15.12:0/2986906645 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8320041020 con 0x7f832c1016e0 2024-03-17T23:57:09.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.239+0000 7f832a000640 1 -- 172.21.15.12:0/2986906645 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8320034cd0 con 0x7f832c1016e0 2024-03-17T23:57:09.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.240+0000 7f8332c00640 1 -- 172.21.15.12:0/2986906645 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f832c1a1a90 con 0x7f832c1016e0 2024-03-17T23:57:09.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.240+0000 7f832a000640 1 -- 172.21.15.12:0/2986906645 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8320039040 con 0x7f832c1016e0 2024-03-17T23:57:09.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.240+0000 7f8332c00640 1 -- 172.21.15.12:0/2986906645 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f832c1a2000 con 0x7f832c1016e0 2024-03-17T23:57:09.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.243+0000 7f832a000640 1 -- 172.21.15.12:0/2986906645 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8320044020 con 0x7f832c1016e0 2024-03-17T23:57:09.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.243+0000 7f8336fd8740 1 Processor -- start 2024-03-17T23:57:09.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.243+0000 7f8336fd8740 1 -- start start 2024-03-17T23:57:09.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.243+0000 7f8336fd8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86d340 0x55a16e86d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.243+0000 7f8336fd8740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 0x55a16e872830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.243+0000 7f8336fd8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e872ea0 0x55a16e875330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.243+0000 7f8336fd8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a16e7d1c50 con 0x55a16e86d340 2024-03-17T23:57:09.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.243+0000 7f8336fd8740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a16e77a9d0 con 0x55a16e86dd30 2024-03-17T23:57:09.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.243+0000 7f8336fd8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a16e7cdcf0 con 0x55a16e872ea0 2024-03-17T23:57:09.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.244+0000 7f8331800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 0x55a16e872830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.244+0000 7f8330e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e872ea0 0x55a16e875330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.244+0000 7f8332200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86d340 0x55a16e86d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.244+0000 7f8331800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 0x55a16e872830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56892/0 (socket says 172.21.15.12:56892) 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.244+0000 7f8331800640 1 -- 172.21.15.12:0/3116142742 learned_addr learned my addr 172.21.15.12:0/3116142742 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.244+0000 7f8331800640 1 -- 172.21.15.12:0/3116142742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e872ea0 msgr2=0x55a16e875330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.244+0000 7f8331800640 1 --2- 172.21.15.12:0/3116142742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e872ea0 0x55a16e875330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.244+0000 7f8331800640 1 -- 172.21.15.12:0/3116142742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86d340 msgr2=0x55a16e86d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.244+0000 7f8331800640 1 --2- 172.21.15.12:0/3116142742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86d340 0x55a16e86d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.244+0000 7f8331800640 1 -- 172.21.15.12:0/3116142742 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a16e794f10 con 0x55a16e86dd30 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.244+0000 7f8330e00640 1 --2- 172.21.15.12:0/3116142742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e872ea0 0x55a16e875330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.245+0000 7f8331800640 1 --2- 172.21.15.12:0/3116142742 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 0x55a16e872830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f83200f4c40 tx=0x7f83200f4d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e9ba947009fab2a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.245+0000 7f831ea00640 1 -- 172.21.15.12:0/3116142742 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83200f9d10 con 0x55a16e86dd30 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.245+0000 7f831ea00640 1 -- 172.21.15.12:0/3116142742 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83201038e0 con 0x55a16e86dd30 2024-03-17T23:57:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.245+0000 7f831ea00640 1 -- 172.21.15.12:0/3116142742 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83200fb840 con 0x55a16e86dd30 2024-03-17T23:57:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.245+0000 7f832a000640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83140a2880 0x7f83140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.245+0000 7f832a000640 1 -- 172.21.15.12:0/2986906645 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f83200f0ca0 con 0x7f832c1016e0 2024-03-17T23:57:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.245+0000 7f8332200640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83140a2880 0x7f83140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.245+0000 7f8336fd8740 1 -- 172.21.15.12:0/3116142742 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 msgr2=0x55a16e872830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.245+0000 7f8336fd8740 1 --2- 172.21.15.12:0/3116142742 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 0x55a16e872830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f83200f4c40 tx=0x7f83200f4d30 comp rx=0 tx=0).stop 2024-03-17T23:57:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.246+0000 7f8336fd8740 1 -- 172.21.15.12:0/3116142742 shutdown_connections 2024-03-17T23:57:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.246+0000 7f8336fd8740 1 --2- 172.21.15.12:0/3116142742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e872ea0 0x55a16e875330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.246+0000 7f8336fd8740 1 --2- 172.21.15.12:0/3116142742 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 0x55a16e872830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.246+0000 7f8336fd8740 1 --2- 172.21.15.12:0/3116142742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86d340 0x55a16e86d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.246+0000 7f8336fd8740 1 -- 172.21.15.12:0/3116142742 >> 172.21.15.12:0/3116142742 conn(0x55a16e8688b0 msgr2=0x55a16e868cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.246+0000 7f8336fd8740 1 -- 172.21.15.12:0/3116142742 shutdown_connections 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.246+0000 7f8336fd8740 1 -- 172.21.15.12:0/3116142742 wait complete. 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.246+0000 7f8332200640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83140a2880 0x7f83140a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x55a16e86e590 tx=0x7f8324005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.247+0000 7f8336fd8740 1 -- start start 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.247+0000 7f8336fd8740 1 Processor -- start 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.247+0000 7f8336fd8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 0x55a16e877240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.247+0000 7f8336fd8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86dd30 0x55a16e877780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.247+0000 7f8336fd8740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e872ea0 0x55a16e87e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8336fd8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a16e875c10 con 0x55a16e86dd30 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8336fd8740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a16e78de20 con 0x55a16e872ea0 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8336fd8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a16e875d90 con 0x55a16e86d340 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8336fd8740 10 client.0 _ll_register_callbacks cb 0x55a16e78d7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8332200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e872ea0 0x55a16e87e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8331800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 0x55a16e877240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8330e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86dd30 0x55a16e877780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8331800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 0x55a16e877240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53678/0 (socket says 172.21.15.12:53678) 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8332200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e872ea0 0x55a16e87e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56906/0 (socket says 172.21.15.12:56906) 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8331800640 1 -- 172.21.15.12:0/3285181245 learned_addr learned my addr 172.21.15.12:0/3285181245 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8331800640 1 -- 172.21.15.12:0/3285181245 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e872ea0 msgr2=0x55a16e87e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8331800640 1 --2- 172.21.15.12:0/3285181245 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e872ea0 0x55a16e87e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8331800640 1 -- 172.21.15.12:0/3285181245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86dd30 msgr2=0x55a16e877780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8331800640 1 --2- 172.21.15.12:0/3285181245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86dd30 0x55a16e877780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8331800640 1 -- 172.21.15.12:0/3285181245 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a16e4e62d0 con 0x55a16e86d340 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.248+0000 7f8331800640 1 --2- 172.21.15.12:0/3285181245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 0x55a16e877240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f83200b96c0 tx=0x7f83200095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5da900f30e58478f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.249+0000 7f8313400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.249+0000 7f8313400640 1 -- 172.21.15.12:0/3285181245 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83200b9500 con 0x55a16e86d340 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.249+0000 7f8313400640 1 -- 172.21.15.12:0/3285181245 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f83200dd0f0 con 0x55a16e86d340 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.249+0000 7f8336fd8740 1 -- 172.21.15.12:0/3285181245 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83201038e0 con 0x55a16e86d340 2024-03-17T23:57:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.249+0000 7f8313400640 1 -- 172.21.15.12:0/3285181245 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f83200bcbc0 con 0x55a16e86d340 2024-03-17T23:57:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.249+0000 7f8336fd8740 10 client.25195 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.249+0000 7f8336fd8740 1 -- 172.21.15.12:0/3285181245 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a16e5b06b0 con 0x55a16e86d340 2024-03-17T23:57:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.249+0000 7f8313400640 1 -- 172.21.15.12:0/3285181245 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83201038e0 con 0x55a16e86d340 2024-03-17T23:57:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.249+0000 7f8313400640 1 -- 172.21.15.12:0/3285181245 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f832003c030 con 0x55a16e86d340 2024-03-17T23:57:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.250+0000 7f8336fd8740 10 client.25195 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.250+0000 7f8336fd8740 10 client.25195 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-17T23:57:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.250+0000 7f8336fd8740 20 client.25195 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.250+0000 7f8336fd8740 1 --2- 172.21.15.12:0/3285181245 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x55a16e883c80 0x55a16e8850f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.250+0000 7f8336fd8740 4 client.25195 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:57:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.250+0000 7f8336fd8740 1 -- 172.21.15.12:0/3285181245 --> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a16e82f2b0 con 0x55a16e883c80 2024-03-17T23:57:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.250+0000 7f8330e00640 1 --2- 172.21.15.12:0/3285181245 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x55a16e883c80 0x55a16e8850f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.251+0000 7f8330e00640 1 --2- 172.21.15.12:0/3285181245 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x55a16e883c80 0x55a16e8850f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=88f8f7ca7ec69814 server_cookie=61fff897d0350f0f in_seq=0 out_seq=0 2024-03-17T23:57:09.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.251+0000 7f8313400640 10 client.25195 ms_handle_connect on v2:172.21.15.12:6840/2073607248 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.252+0000 7f8313400640 1 -- 172.21.15.12:0/3285181245 <== mds.1 v2:172.21.15.12:6840/2073607248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a16e82f2b0 con 0x55a16e883c80 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.252+0000 7f8313400640 10 client.25195 handle_command_reply: tid=0 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.252+0000 7f8336fd8740 1 client.25195 shutdown 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.252+0000 7f8336fd8740 1 -- 172.21.15.12:0/3285181245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 msgr2=0x55a16e877240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.252+0000 7f8336fd8740 1 --2- 172.21.15.12:0/3285181245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 0x55a16e877240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f83200b96c0 tx=0x7f83200095c0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.252+0000 7f8336fd8740 1 -- 172.21.15.12:0/3285181245 shutdown_connections 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.252+0000 7f8336fd8740 1 --2- 172.21.15.12:0/3285181245 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e872ea0 0x55a16e87e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.252+0000 7f8336fd8740 1 --2- 172.21.15.12:0/3285181245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86dd30 0x55a16e877780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.252+0000 7f8336fd8740 1 --2- 172.21.15.12:0/3285181245 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x55a16e883c80 0x55a16e8850f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.252+0000 7f8336fd8740 1 --2- 172.21.15.12:0/3285181245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 0x55a16e877240 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.252+0000 7f8313400640 0 client.25195 ms_handle_reset on v2:172.21.15.12:6840/2073607248 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.252+0000 7f8336fd8740 1 -- 172.21.15.12:0/3285181245 >> 172.21.15.12:0/3285181245 conn(0x55a16e8688b0 msgr2=0x55a16e867a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.253+0000 7f8336fd8740 1 -- 172.21.15.12:0/3285181245 shutdown_connections 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.253+0000 7f8336fd8740 1 -- 172.21.15.12:0/3285181245 wait complete. 2024-03-17T23:57:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.253+0000 7f8336fd8740 20 client.25195 trim_cache size 0 max 16384 2024-03-17T23:57:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8336fd8740 1 Processor -- start 2024-03-17T23:57:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8336fd8740 1 -- start start 2024-03-17T23:57:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8336fd8740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86d340 0x55a16e876e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8336fd8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86dd30 0x55a16e87e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8336fd8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e872ea0 0x55a16e877390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8336fd8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a16e883430 con 0x55a16e86dd30 2024-03-17T23:57:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8336fd8740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a16e883b00 con 0x55a16e86d340 2024-03-17T23:57:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8336fd8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a16e8841d0 con 0x55a16e872ea0 2024-03-17T23:57:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8330e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86dd30 0x55a16e87e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8331800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86d340 0x55a16e876e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8330e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86dd30 0x55a16e87e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47188/0 (socket says 172.21.15.12:47188) 2024-03-17T23:57:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8331800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86d340 0x55a16e876e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56920/0 (socket says 172.21.15.12:56920) 2024-03-17T23:57:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8330e00640 1 -- 172.21.15.12:0/1114180842 learned_addr learned my addr 172.21.15.12:0/1114180842 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.275+0000 7f8332200640 1 --2- 172.21.15.12:0/1114180842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e872ea0 0x55a16e877390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.276+0000 7f8331800640 1 -- 172.21.15.12:0/1114180842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e872ea0 msgr2=0x55a16e877390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.276+0000 7f8331800640 1 --2- 172.21.15.12:0/1114180842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e872ea0 0x55a16e877390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.276+0000 7f8331800640 1 -- 172.21.15.12:0/1114180842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86dd30 msgr2=0x55a16e87e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.276+0000 7f8331800640 1 --2- 172.21.15.12:0/1114180842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86dd30 0x55a16e87e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.276+0000 7f8331800640 1 -- 172.21.15.12:0/1114180842 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a16e86e110 con 0x55a16e86d340 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.276+0000 7f8330e00640 1 --2- 172.21.15.12:0/1114180842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86dd30 0x55a16e87e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.276+0000 7f8331800640 1 --2- 172.21.15.12:0/1114180842 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86d340 0x55a16e876e50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f832002c700 tx=0x7f83200f4d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72e839628c7fbc15 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.276+0000 7f831e000640 1 -- 172.21.15.12:0/1114180842 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83200bcda0 con 0x55a16e86d340 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.276+0000 7f831e000640 1 -- 172.21.15.12:0/1114180842 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83200bd960 con 0x55a16e86d340 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.276+0000 7f831e000640 1 -- 172.21.15.12:0/1114180842 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83200bd030 con 0x55a16e86d340 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.277+0000 7f8336fd8740 1 -- 172.21.15.12:0/1114180842 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86d340 msgr2=0x55a16e876e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.277+0000 7f8336fd8740 1 --2- 172.21.15.12:0/1114180842 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86d340 0x55a16e876e50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f832002c700 tx=0x7f83200f4d30 comp rx=0 tx=0).stop 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.277+0000 7f8336fd8740 1 -- 172.21.15.12:0/1114180842 shutdown_connections 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.277+0000 7f8336fd8740 1 --2- 172.21.15.12:0/1114180842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e872ea0 0x55a16e877390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.277+0000 7f8336fd8740 1 --2- 172.21.15.12:0/1114180842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e86dd30 0x55a16e87e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.277+0000 7f8336fd8740 1 --2- 172.21.15.12:0/1114180842 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86d340 0x55a16e876e50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.277+0000 7f8336fd8740 1 -- 172.21.15.12:0/1114180842 >> 172.21.15.12:0/1114180842 conn(0x55a16e8688b0 msgr2=0x55a16e863900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.277+0000 7f8336fd8740 1 -- 172.21.15.12:0/1114180842 shutdown_connections 2024-03-17T23:57:09.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.277+0000 7f8336fd8740 1 -- 172.21.15.12:0/1114180842 wait complete. 2024-03-17T23:57:09.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.277+0000 7f8336fd8740 1 -- start start 2024-03-17T23:57:09.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8336fd8740 1 Processor -- start 2024-03-17T23:57:09.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8336fd8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 0x55a16e87c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8336fd8740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 0x55a16e8e59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8336fd8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e872ea0 0x55a16e8e7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8336fd8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a16e884f70 con 0x55a16e872ea0 2024-03-17T23:57:09.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8336fd8740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a16e8848a0 con 0x55a16e86dd30 2024-03-17T23:57:09.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8336fd8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a16e885640 con 0x55a16e86d340 2024-03-17T23:57:09.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8336fd8740 10 client.0 _ll_register_callbacks cb 0x7f83200f4a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8332200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e872ea0 0x55a16e8e7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8330e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 0x55a16e8e59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8331800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 0x55a16e87c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8331800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 0x55a16e87c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53688/0 (socket says 172.21.15.12:53688) 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8331800640 1 -- 172.21.15.12:0/1270788474 learned_addr learned my addr 172.21.15.12:0/1270788474 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8330e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 0x55a16e8e59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56922/0 (socket says 172.21.15.12:56922) 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.278+0000 7f8332200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e872ea0 0x55a16e8e7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47198/0 (socket says 172.21.15.12:47198) 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.279+0000 7f8331800640 1 -- 172.21.15.12:0/1270788474 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 msgr2=0x55a16e8e59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.279+0000 7f8331800640 1 --2- 172.21.15.12:0/1270788474 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 0x55a16e8e59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.279+0000 7f8331800640 1 -- 172.21.15.12:0/1270788474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e872ea0 msgr2=0x55a16e8e7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.279+0000 7f8331800640 1 --2- 172.21.15.12:0/1270788474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e872ea0 0x55a16e8e7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.279+0000 7f8331800640 1 -- 172.21.15.12:0/1270788474 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a16e8e8560 con 0x55a16e86d340 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.279+0000 7f8330e00640 1 --2- 172.21.15.12:0/1270788474 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 0x55a16e8e59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.279+0000 7f8332200640 1 --2- 172.21.15.12:0/1270788474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e872ea0 0x55a16e8e7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:09.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.279+0000 7f8331800640 1 --2- 172.21.15.12:0/1270788474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 0x55a16e87c6d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f83200bb510 tx=0x7f83200095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3d3ccb97cff737b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.279+0000 7f8313400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.279+0000 7f8313400640 1 -- 172.21.15.12:0/1270788474 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83200bd030 con 0x55a16e86d340 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.279+0000 7f8313400640 1 -- 172.21.15.12:0/1270788474 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f83200dd0b0 con 0x55a16e86d340 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.279+0000 7f8336fd8740 1 -- 172.21.15.12:0/1270788474 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83200bd960 con 0x55a16e86d340 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.279+0000 7f8313400640 1 -- 172.21.15.12:0/1270788474 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f83200bcda0 con 0x55a16e86d340 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.280+0000 7f8336fd8740 10 client.25201 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.280+0000 7f8336fd8740 1 -- 172.21.15.12:0/1270788474 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a16e8e8560 con 0x55a16e86d340 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.280+0000 7f8313400640 1 -- 172.21.15.12:0/1270788474 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83200bd960 con 0x55a16e86d340 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.280+0000 7f8313400640 1 -- 172.21.15.12:0/1270788474 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8320003a70 con 0x55a16e86d340 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.280+0000 7f8336fd8740 10 client.25201 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.280+0000 7f8336fd8740 10 client.25201 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.280+0000 7f8336fd8740 20 client.25201 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.280+0000 7f8336fd8740 1 --2- 172.21.15.12:0/1270788474 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x55a16e8edd00 0x55a16e8eca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.280+0000 7f8336fd8740 4 client.25201 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:57:09.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.280+0000 7f8336fd8740 1 -- 172.21.15.12:0/1270788474 --> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] -- command(tid 0: {"prefix": "status"}) -- 0x55a16e7df190 con 0x55a16e8edd00 2024-03-17T23:57:09.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.280+0000 7f8330e00640 1 --2- 172.21.15.12:0/1270788474 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x55a16e8edd00 0x55a16e8eca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.281+0000 7f8330e00640 1 --2- 172.21.15.12:0/1270788474 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x55a16e8edd00 0x55a16e8eca10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5530ef3faa9404d6 server_cookie=f06260bb53f3b169 in_seq=0 out_seq=0 2024-03-17T23:57:09.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.281+0000 7f8313400640 10 client.25201 ms_handle_connect on v2:172.21.15.12:6840/2073607248 2024-03-17T23:57:09.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.281+0000 7f8313400640 1 -- 172.21.15.12:0/1270788474 <== mds.1 v2:172.21.15.12:6840/2073607248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55a16e7df190 con 0x55a16e8edd00 2024-03-17T23:57:09.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.281+0000 7f8313400640 10 client.25201 handle_command_reply: tid=0 2024-03-17T23:57:09.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.281+0000 7f8336fd8740 1 client.25201 shutdown 2024-03-17T23:57:09.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.281+0000 7f8336fd8740 1 -- 172.21.15.12:0/1270788474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 msgr2=0x55a16e87c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.282+0000 7f8336fd8740 1 --2- 172.21.15.12:0/1270788474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 0x55a16e87c6d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f83200bb510 tx=0x7f83200095c0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.282+0000 7f8336fd8740 1 -- 172.21.15.12:0/1270788474 shutdown_connections 2024-03-17T23:57:09.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.282+0000 7f8336fd8740 1 --2- 172.21.15.12:0/1270788474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a16e872ea0 0x55a16e8e7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.282+0000 7f8336fd8740 1 --2- 172.21.15.12:0/1270788474 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a16e86dd30 0x55a16e8e59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.282+0000 7f8336fd8740 1 --2- 172.21.15.12:0/1270788474 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x55a16e8edd00 0x55a16e8eca10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.282+0000 7f8336fd8740 1 --2- 172.21.15.12:0/1270788474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a16e86d340 0x55a16e87c6d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.282+0000 7f8336fd8740 1 -- 172.21.15.12:0/1270788474 >> 172.21.15.12:0/1270788474 conn(0x55a16e8688b0 msgr2=0x55a16e8671e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.282+0000 7f8313400640 0 client.25201 ms_handle_reset on v2:172.21.15.12:6840/2073607248 2024-03-17T23:57:09.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.282+0000 7f8336fd8740 1 -- 172.21.15.12:0/1270788474 shutdown_connections 2024-03-17T23:57:09.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.282+0000 7f8336fd8740 1 -- 172.21.15.12:0/1270788474 wait complete. 2024-03-17T23:57:09.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.282+0000 7f8336fd8740 20 client.25201 trim_cache size 0 max 16384 2024-03-17T23:57:09.287 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-17T23:57:09.287 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:57:09.287 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 1, 2024-03-17T23:57:09.287 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24575, 2024-03-17T23:57:09.288 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-17T23:57:09.288 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-17T23:57:09.288 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-17T23:57:09.288 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 105.307729804, 2024-03-17T23:57:09.288 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 28, 2024-03-17T23:57:09.288 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 94, 2024-03-17T23:57:09.288 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 90, 2024-03-17T23:57:09.288 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 141.878379924 2024-03-17T23:57:09.288 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-17T23:57:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.283+0000 7f8332c00640 1 -- 172.21.15.12:0/2986906645 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83140a2880 msgr2=0x7f83140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.283+0000 7f8332c00640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83140a2880 0x7f83140a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x55a16e86e590 tx=0x7f8324005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.283+0000 7f8332c00640 1 -- 172.21.15.12:0/2986906645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f832c1016e0 msgr2=0x7f832c075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.283+0000 7f8332c00640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f832c1016e0 0x7f832c075590 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8320000f80 tx=0x7f8320032ec0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.283+0000 7f8332c00640 1 -- 172.21.15.12:0/2986906645 shutdown_connections 2024-03-17T23:57:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.283+0000 7f8332c00640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f83140a2880 0x7f83140a4d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.283+0000 7f8332c00640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f832c103290 0x7f832c074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.283+0000 7f8332c00640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f832c1028d0 0x7f832c073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.283+0000 7f8332c00640 1 --2- 172.21.15.12:0/2986906645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f832c1016e0 0x7f832c075590 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.283+0000 7f8332c00640 1 -- 172.21.15.12:0/2986906645 >> 172.21.15.12:0/2986906645 conn(0x7f832c0fcc50 msgr2=0x7f832c105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.283+0000 7f8332c00640 1 -- 172.21.15.12:0/2986906645 shutdown_connections 2024-03-17T23:57:09.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.283+0000 7f8332c00640 1 -- 172.21.15.12:0/2986906645 wait complete. 2024-03-17T23:57:09.300 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-17T23:57:09.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.512+0000 7feec0a00640 1 -- 172.21.15.12:0/2846136238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feebc104ad0 msgr2=0x7feebc104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.512+0000 7feec0a00640 1 --2- 172.21.15.12:0/2846136238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feebc104ad0 0x7feebc104f50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7feea4009900 tx=0x7feea402bc50 comp rx=0 tx=0).stop 2024-03-17T23:57:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.512+0000 7feec0a00640 1 -- 172.21.15.12:0/2846136238 shutdown_connections 2024-03-17T23:57:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.512+0000 7feec0a00640 1 --2- 172.21.15.12:0/2846136238 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feebc105490 0x7feebc10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.512+0000 7feec0a00640 1 --2- 172.21.15.12:0/2846136238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feebc104ad0 0x7feebc104f50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.512+0000 7feec0a00640 1 --2- 172.21.15.12:0/2846136238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc1038e0 0x7feebc103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.512+0000 7feec0a00640 1 -- 172.21.15.12:0/2846136238 >> 172.21.15.12:0/2846136238 conn(0x7feebc0fee50 msgr2=0x7feebc101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.512+0000 7feec0a00640 1 -- 172.21.15.12:0/2846136238 shutdown_connections 2024-03-17T23:57:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.512+0000 7feec0a00640 1 -- 172.21.15.12:0/2846136238 wait complete. 2024-03-17T23:57:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.513+0000 7feec0a00640 1 Processor -- start 2024-03-17T23:57:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.513+0000 7feec0a00640 1 -- start start 2024-03-17T23:57:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.513+0000 7feec0a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feebc1038e0 0x7feebc199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.513+0000 7feec0a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feebc104ad0 0x7feebc19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.513+0000 7feec0a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc105490 0x7feebc1a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.513+0000 7feec0a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feebc10f540 con 0x7feebc105490 2024-03-17T23:57:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.513+0000 7feec0a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7feebc10f3c0 con 0x7feebc104ad0 2024-03-17T23:57:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.513+0000 7feec0a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7feebc10f6c0 con 0x7feebc1038e0 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.514+0000 7feeb3e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feebc104ad0 0x7feebc19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.514+0000 7feeba800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feebc1038e0 0x7feebc199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.514+0000 7feeb3e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feebc104ad0 0x7feebc19a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56940/0 (socket says 172.21.15.12:56940) 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.514+0000 7feebb200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc105490 0x7feebc1a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.514+0000 7feeb3e00640 1 -- 172.21.15.12:0/220178251 learned_addr learned my addr 172.21.15.12:0/220178251 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.514+0000 7feeb3e00640 1 -- 172.21.15.12:0/220178251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feebc1038e0 msgr2=0x7feebc199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.514+0000 7feeb3e00640 1 --2- 172.21.15.12:0/220178251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feebc1038e0 0x7feebc199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.514+0000 7feeb3e00640 1 -- 172.21.15.12:0/220178251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc105490 msgr2=0x7feebc1a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.514+0000 7feeb3e00640 1 --2- 172.21.15.12:0/220178251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc105490 0x7feebc1a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.514+0000 7feeb3e00640 1 -- 172.21.15.12:0/220178251 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feebc1a18c0 con 0x7feebc104ad0 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.514+0000 7feebb200640 1 --2- 172.21.15.12:0/220178251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc105490 0x7feebc1a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.514+0000 7feeb3e00640 1 --2- 172.21.15.12:0/220178251 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feebc104ad0 0x7feebc19a1c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7feea4009230 tx=0x7feea402c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.515+0000 7feeb3400640 1 -- 172.21.15.12:0/220178251 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4042020 con 0x7feebc104ad0 2024-03-17T23:57:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.515+0000 7feeb3400640 1 -- 172.21.15.12:0/220178251 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea4004450 con 0x7feebc104ad0 2024-03-17T23:57:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.515+0000 7feeb3400640 1 -- 172.21.15.12:0/220178251 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4003e20 con 0x7feebc104ad0 2024-03-17T23:57:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.515+0000 7feec0a00640 1 -- 172.21.15.12:0/220178251 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feebc1a1b50 con 0x7feebc104ad0 2024-03-17T23:57:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.515+0000 7feec0a00640 1 -- 172.21.15.12:0/220178251 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feebc1a20f0 con 0x7feebc104ad0 2024-03-17T23:57:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.519+0000 7feec420b740 1 Processor -- start 2024-03-17T23:57:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.519+0000 7feec420b740 1 -- start start 2024-03-17T23:57:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.520+0000 7feeb3400640 1 -- 172.21.15.12:0/220178251 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7feea4045020 con 0x7feebc104ad0 2024-03-17T23:57:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.520+0000 7feec420b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915d340 0x55a30915d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.520+0000 7feec420b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a30915dd30 0x55a309162830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.520+0000 7feec420b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a309162ea0 0x55a309165330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.520+0000 7feec420b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a3090c1c50 con 0x55a30915d340 2024-03-17T23:57:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.520+0000 7feec420b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a30906a9d0 con 0x55a30915dd30 2024-03-17T23:57:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.520+0000 7feec420b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a3090bdcf0 con 0x55a309162ea0 2024-03-17T23:57:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.520+0000 7feebb200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915d340 0x55a30915d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.520+0000 7feeb3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a309162ea0 0x55a309165330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.520+0000 7feeba800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a30915dd30 0x55a309162830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.520+0000 7feeba800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a30915dd30 0x55a309162830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56948/0 (socket says 172.21.15.12:56948) 2024-03-17T23:57:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.520+0000 7feeb3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a309162ea0 0x55a309165330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53708/0 (socket says 172.21.15.12:53708) 2024-03-17T23:57:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.520+0000 7feeba800640 1 -- 172.21.15.12:0/3040202953 learned_addr learned my addr 172.21.15.12:0/3040202953 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:09.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.521+0000 7feeba800640 1 -- 172.21.15.12:0/3040202953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a309162ea0 msgr2=0x55a309165330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.521+0000 7feeba800640 1 --2- 172.21.15.12:0/3040202953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a309162ea0 0x55a309165330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.521+0000 7feeba800640 1 -- 172.21.15.12:0/3040202953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915d340 msgr2=0x55a30915d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.521+0000 7feeba800640 1 --2- 172.21.15.12:0/3040202953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915d340 0x55a30915d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.521+0000 7feeba800640 1 -- 172.21.15.12:0/3040202953 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a309084f10 con 0x55a30915dd30 2024-03-17T23:57:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.521+0000 7feeb3e00640 1 --2- 172.21.15.12:0/3040202953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a309162ea0 0x55a309165330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.521+0000 7feeba800640 1 --2- 172.21.15.12:0/3040202953 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a30915dd30 0x55a309162830 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7feeac009c60 tx=0x7feeac011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70444f65e861e0e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.521+0000 7feeab400640 1 -- 172.21.15.12:0/3040202953 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac004220 con 0x55a30915dd30 2024-03-17T23:57:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.521+0000 7feeb3400640 1 --2- 172.21.15.12:0/220178251 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7feea00a2880 0x7feea00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.521+0000 7feeab400640 1 -- 172.21.15.12:0/3040202953 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeac0043c0 con 0x55a30915dd30 2024-03-17T23:57:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.521+0000 7feeab400640 1 -- 172.21.15.12:0/3040202953 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac004e10 con 0x55a30915dd30 2024-03-17T23:57:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.521+0000 7feeb3400640 1 -- 172.21.15.12:0/220178251 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7feea40f0d70 con 0x7feebc104ad0 2024-03-17T23:57:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.522+0000 7feebb200640 1 --2- 172.21.15.12:0/220178251 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7feea00a2880 0x7feea00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.522+0000 7feec420b740 1 -- 172.21.15.12:0/3040202953 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a30915dd30 msgr2=0x55a309162830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.522+0000 7feec420b740 1 --2- 172.21.15.12:0/3040202953 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a30915dd30 0x55a309162830 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7feeac009c60 tx=0x7feeac011810 comp rx=0 tx=0).stop 2024-03-17T23:57:09.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.522+0000 7feec420b740 1 -- 172.21.15.12:0/3040202953 shutdown_connections 2024-03-17T23:57:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.522+0000 7feec420b740 1 --2- 172.21.15.12:0/3040202953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a309162ea0 0x55a309165330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.522+0000 7feec420b740 1 --2- 172.21.15.12:0/3040202953 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a30915dd30 0x55a309162830 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.522+0000 7feec420b740 1 --2- 172.21.15.12:0/3040202953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915d340 0x55a30915d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.522+0000 7feec420b740 1 -- 172.21.15.12:0/3040202953 >> 172.21.15.12:0/3040202953 conn(0x55a3091588b0 msgr2=0x55a309158cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.522+0000 7feec420b740 1 -- 172.21.15.12:0/3040202953 shutdown_connections 2024-03-17T23:57:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.522+0000 7feebb200640 1 --2- 172.21.15.12:0/220178251 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7feea00a2880 0x7feea00a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55a30915e590 tx=0x7feeb4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.522+0000 7feec420b740 1 -- 172.21.15.12:0/3040202953 wait complete. 2024-03-17T23:57:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.523+0000 7feec420b740 1 -- start start 2024-03-17T23:57:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.523+0000 7feec420b740 1 Processor -- start 2024-03-17T23:57:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feec420b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 0x55a3091674d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feec420b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915dd30 0x55a309167a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feec420b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a309162ea0 0x55a30916ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feec420b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a309165e50 con 0x55a30915dd30 2024-03-17T23:57:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feec420b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a30907de20 con 0x55a309162ea0 2024-03-17T23:57:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feec420b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a309165fd0 con 0x55a30915d340 2024-03-17T23:57:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feec420b740 10 client.0 _ll_register_callbacks cb 0x55a30907d7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feeba800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 0x55a3091674d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feebb200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a309162ea0 0x55a30916ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feeb3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915dd30 0x55a309167a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feeba800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 0x55a3091674d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53724/0 (socket says 172.21.15.12:53724) 2024-03-17T23:57:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feeba800640 1 -- 172.21.15.12:0/3233630696 learned_addr learned my addr 172.21.15.12:0/3233630696 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:09.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feeba800640 1 -- 172.21.15.12:0/3233630696 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a309162ea0 msgr2=0x55a30916ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feeba800640 1 --2- 172.21.15.12:0/3233630696 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a309162ea0 0x55a30916ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feeba800640 1 -- 172.21.15.12:0/3233630696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915dd30 msgr2=0x55a309167a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feeba800640 1 --2- 172.21.15.12:0/3233630696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915dd30 0x55a309167a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feeba800640 1 -- 172.21.15.12:0/3233630696 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a308dd62d0 con 0x55a30915d340 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.524+0000 7feebb200640 1 --2- 172.21.15.12:0/3233630696 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a309162ea0 0x55a30916ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.525+0000 7feeba800640 1 --2- 172.21.15.12:0/3233630696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 0x55a3091674d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feeac011d10 tx=0x7feeac005190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=404fb1a35ef294a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.525+0000 7fee9fe00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.525+0000 7fee9fe00640 1 -- 172.21.15.12:0/3233630696 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac027030 con 0x55a30915d340 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.525+0000 7fee9fe00640 1 -- 172.21.15.12:0/3233630696 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7feeac005380 con 0x55a30915d340 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.525+0000 7feec420b740 1 -- 172.21.15.12:0/3233630696 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feeac0043c0 con 0x55a30915d340 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.525+0000 7fee9fe00640 1 -- 172.21.15.12:0/3233630696 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7feeac034050 con 0x55a30915d340 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.525+0000 7feec420b740 10 client.25213 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.525+0000 7feec420b740 1 -- 172.21.15.12:0/3233630696 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a308ea06b0 con 0x55a30915d340 2024-03-17T23:57:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.526+0000 7fee9fe00640 1 -- 172.21.15.12:0/3233630696 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeac0043c0 con 0x55a30915d340 2024-03-17T23:57:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.526+0000 7fee9fe00640 1 -- 172.21.15.12:0/3233630696 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7feeac0199c0 con 0x55a30915d340 2024-03-17T23:57:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.526+0000 7feec420b740 10 client.25213 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.526+0000 7feec420b740 10 client.25213 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-17T23:57:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.526+0000 7feec420b740 20 client.25213 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.526+0000 7feec420b740 1 --2- 172.21.15.12:0/3233630696 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55a309173fd0 0x55a309175440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.526+0000 7feec420b740 4 client.25213 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:57:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.526+0000 7feec420b740 1 -- 172.21.15.12:0/3233630696 --> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a30911f2b0 con 0x55a309173fd0 2024-03-17T23:57:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.526+0000 7feeb3e00640 1 --2- 172.21.15.12:0/3233630696 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55a309173fd0 0x55a309175440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.527+0000 7feeb3e00640 1 --2- 172.21.15.12:0/3233630696 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55a309173fd0 0x55a309175440 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7df704fdd142dc server_cookie=96987b720f416af8 in_seq=0 out_seq=0 2024-03-17T23:57:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.527+0000 7fee9fe00640 10 client.25213 ms_handle_connect on v2:172.21.15.79:6838/2478742711 2024-03-17T23:57:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7fee9fe00640 1 -- 172.21.15.12:0/3233630696 <== mds.0 v2:172.21.15.79:6838/2478742711 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a30911f2b0 con 0x55a309173fd0 2024-03-17T23:57:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7fee9fe00640 10 client.25213 handle_command_reply: tid=0 2024-03-17T23:57:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7feec420b740 1 client.25213 shutdown 2024-03-17T23:57:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7feec420b740 1 -- 172.21.15.12:0/3233630696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 msgr2=0x55a3091674d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7feec420b740 1 --2- 172.21.15.12:0/3233630696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 0x55a3091674d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feeac011d10 tx=0x7feeac005190 comp rx=0 tx=0).stop 2024-03-17T23:57:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7feec420b740 1 -- 172.21.15.12:0/3233630696 shutdown_connections 2024-03-17T23:57:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7feec420b740 1 --2- 172.21.15.12:0/3233630696 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55a309173fd0 0x55a309175440 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7feec420b740 1 --2- 172.21.15.12:0/3233630696 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a309162ea0 0x55a30916ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7feec420b740 1 --2- 172.21.15.12:0/3233630696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915dd30 0x55a309167a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7fee9fe00640 0 client.25213 ms_handle_reset on v2:172.21.15.79:6838/2478742711 2024-03-17T23:57:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7feec420b740 1 --2- 172.21.15.12:0/3233630696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 0x55a3091674d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7feec420b740 1 -- 172.21.15.12:0/3233630696 >> 172.21.15.12:0/3233630696 conn(0x55a3091588b0 msgr2=0x55a309157a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7feec420b740 1 -- 172.21.15.12:0/3233630696 shutdown_connections 2024-03-17T23:57:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7feec420b740 1 -- 172.21.15.12:0/3233630696 wait complete. 2024-03-17T23:57:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.528+0000 7feec420b740 20 client.25213 trim_cache size 0 max 16384 2024-03-17T23:57:09.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.560+0000 7feec420b740 1 Processor -- start 2024-03-17T23:57:09.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.560+0000 7feec420b740 1 -- start start 2024-03-17T23:57:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feec420b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 0x55a309153900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feec420b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a30915dd30 0x55a30916ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feec420b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a309162ea0 0x55a309158c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feec420b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a309173780 con 0x55a309162ea0 2024-03-17T23:57:09.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feec420b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a309173e50 con 0x55a30915dd30 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feec420b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a309174520 con 0x55a30915d340 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feeba800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 0x55a309153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feeb3e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a30915dd30 0x55a30916ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feeba800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 0x55a309153900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53732/0 (socket says 172.21.15.12:53732) 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feeba800640 1 -- 172.21.15.12:0/3386044227 learned_addr learned my addr 172.21.15.12:0/3386044227 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feebb200640 1 --2- 172.21.15.12:0/3386044227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a309162ea0 0x55a309158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feeba800640 1 -- 172.21.15.12:0/3386044227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a30915dd30 msgr2=0x55a30916ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feeba800640 1 --2- 172.21.15.12:0/3386044227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a30915dd30 0x55a30916ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feeba800640 1 -- 172.21.15.12:0/3386044227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a309162ea0 msgr2=0x55a309158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feeba800640 1 --2- 172.21.15.12:0/3386044227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a309162ea0 0x55a309158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feeba800640 1 -- 172.21.15.12:0/3386044227 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3091581f0 con 0x55a30915d340 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feebb200640 1 --2- 172.21.15.12:0/3386044227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a309162ea0 0x55a309158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.561+0000 7feeb3e00640 1 --2- 172.21.15.12:0/3386044227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a30915dd30 0x55a30916ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.562+0000 7feeba800640 1 --2- 172.21.15.12:0/3386044227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 0x55a309153900 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feeac019f80 tx=0x7feeac011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2341cb32dc3af51 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.562+0000 7feeaaa00640 1 -- 172.21.15.12:0/3386044227 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac027040 con 0x55a30915d340 2024-03-17T23:57:09.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.562+0000 7feeaaa00640 1 -- 172.21.15.12:0/3386044227 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeac02e970 con 0x55a30915d340 2024-03-17T23:57:09.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.562+0000 7feeaaa00640 1 -- 172.21.15.12:0/3386044227 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac01d040 con 0x55a30915d340 2024-03-17T23:57:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.562+0000 7feec420b740 1 -- 172.21.15.12:0/3386044227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 msgr2=0x55a309153900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.562+0000 7feec420b740 1 --2- 172.21.15.12:0/3386044227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 0x55a309153900 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feeac019f80 tx=0x7feeac011810 comp rx=0 tx=0).stop 2024-03-17T23:57:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.562+0000 7feec420b740 1 -- 172.21.15.12:0/3386044227 shutdown_connections 2024-03-17T23:57:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.562+0000 7feec420b740 1 --2- 172.21.15.12:0/3386044227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a309162ea0 0x55a309158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.562+0000 7feec420b740 1 --2- 172.21.15.12:0/3386044227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a30915dd30 0x55a30916ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.562+0000 7feec420b740 1 --2- 172.21.15.12:0/3386044227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915d340 0x55a309153900 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.562+0000 7feec420b740 1 -- 172.21.15.12:0/3386044227 >> 172.21.15.12:0/3386044227 conn(0x55a3091588b0 msgr2=0x55a30915e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.563+0000 7feec420b740 1 -- 172.21.15.12:0/3386044227 shutdown_connections 2024-03-17T23:57:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.563+0000 7feec420b740 1 -- 172.21.15.12:0/3386044227 wait complete. 2024-03-17T23:57:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.563+0000 7feec420b740 1 -- start start 2024-03-17T23:57:09.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.563+0000 7feec420b740 1 Processor -- start 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feec420b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915d340 0x55a30916a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feec420b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915dd30 0x55a30916a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feec420b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a309162ea0 0x55a30916d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feec420b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a3091752c0 con 0x55a30915d340 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feec420b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a309174bf0 con 0x55a309162ea0 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feec420b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a309175990 con 0x55a30915dd30 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feec420b740 10 client.0 _ll_register_callbacks cb 0x7feeac009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feebb200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a309162ea0 0x55a30916d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feeb3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915dd30 0x55a30916a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feebb200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a309162ea0 0x55a30916d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56972/0 (socket says 172.21.15.12:56972) 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feebb200640 1 -- 172.21.15.12:0/3680302504 learned_addr learned my addr 172.21.15.12:0/3680302504 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feeba800640 1 --2- 172.21.15.12:0/3680302504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915d340 0x55a30916a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feebb200640 1 -- 172.21.15.12:0/3680302504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915dd30 msgr2=0x55a30916a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feebb200640 1 --2- 172.21.15.12:0/3680302504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915dd30 0x55a30916a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feebb200640 1 -- 172.21.15.12:0/3680302504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915d340 msgr2=0x55a30916a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feebb200640 1 --2- 172.21.15.12:0/3680302504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915d340 0x55a30916a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.564+0000 7feebb200640 1 -- 172.21.15.12:0/3680302504 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a30916dc70 con 0x55a309162ea0 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.565+0000 7feebb200640 1 --2- 172.21.15.12:0/3680302504 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a309162ea0 0x55a30916d680 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7feeb4011490 tx=0x7feeb4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5f6cd9464914cb server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.565+0000 7fee9fe00640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.565+0000 7fee9fe00640 1 -- 172.21.15.12:0/3680302504 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb402e060 con 0x55a309162ea0 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.565+0000 7feec420b740 1 -- 172.21.15.12:0/3680302504 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feeac02e970 con 0x55a309162ea0 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.565+0000 7fee9fe00640 1 -- 172.21.15.12:0/3680302504 <== mon.2 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7feeb4029420 con 0x55a309162ea0 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.565+0000 7fee9fe00640 1 -- 172.21.15.12:0/3680302504 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7feeb4035050 con 0x55a309162ea0 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.565+0000 7feec420b740 10 client.25238 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.565+0000 7feec420b740 1 -- 172.21.15.12:0/3680302504 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a3091d9e00 con 0x55a309162ea0 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.565+0000 7fee9fe00640 1 -- 172.21.15.12:0/3680302504 <== mon.2 v2:172.21.15.79:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeac02e970 con 0x55a309162ea0 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.566+0000 7fee9fe00640 1 -- 172.21.15.12:0/3680302504 <== mon.2 v2:172.21.15.79:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7feeb401f6c0 con 0x55a309162ea0 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.566+0000 7feec420b740 10 client.25238 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:09.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.566+0000 7feec420b740 10 client.25238 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-17T23:57:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.566+0000 7feec420b740 20 client.25238 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.566+0000 7feec420b740 1 --2- 172.21.15.12:0/3680302504 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55a3091dea90 0x55a3091dd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.566+0000 7feec420b740 4 client.25238 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:57:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.566+0000 7feec420b740 1 -- 172.21.15.12:0/3680302504 --> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] -- command(tid 0: {"prefix": "status"}) -- 0x55a3090cf190 con 0x55a3091dea90 2024-03-17T23:57:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.566+0000 7feeba800640 1 --2- 172.21.15.12:0/3680302504 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55a3091dea90 0x55a3091dd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.566+0000 7feeba800640 1 --2- 172.21.15.12:0/3680302504 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55a3091dea90 0x55a3091dd7a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1e6085122c3a4ad server_cookie=dcfc8e881df48020 in_seq=0 out_seq=0 2024-03-17T23:57:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.566+0000 7fee9fe00640 10 client.25238 ms_handle_connect on v2:172.21.15.79:6838/2478742711 2024-03-17T23:57:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7fee9fe00640 1 -- 172.21.15.12:0/3680302504 <== mds.0 v2:172.21.15.79:6838/2478742711 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a3090cf190 con 0x55a3091dea90 2024-03-17T23:57:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7fee9fe00640 10 client.25238 handle_command_reply: tid=0 2024-03-17T23:57:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7feec420b740 1 client.25238 shutdown 2024-03-17T23:57:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7feec420b740 1 -- 172.21.15.12:0/3680302504 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a309162ea0 msgr2=0x55a30916d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7feec420b740 1 --2- 172.21.15.12:0/3680302504 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a309162ea0 0x55a30916d680 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7feeb4011490 tx=0x7feeb4004a40 comp rx=0 tx=0).stop 2024-03-17T23:57:09.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7feec420b740 1 -- 172.21.15.12:0/3680302504 shutdown_connections 2024-03-17T23:57:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7feec420b740 1 --2- 172.21.15.12:0/3680302504 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55a3091dea90 0x55a3091dd7a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7feec420b740 1 --2- 172.21.15.12:0/3680302504 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a309162ea0 0x55a30916d680 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7feec420b740 1 --2- 172.21.15.12:0/3680302504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a30915dd30 0x55a30916a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7feec420b740 1 --2- 172.21.15.12:0/3680302504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30915d340 0x55a30916a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7feec420b740 1 -- 172.21.15.12:0/3680302504 >> 172.21.15.12:0/3680302504 conn(0x55a3091588b0 msgr2=0x55a309157a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7fee9fe00640 0 client.25238 ms_handle_reset on v2:172.21.15.79:6838/2478742711 2024-03-17T23:57:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7feec420b740 1 -- 172.21.15.12:0/3680302504 shutdown_connections 2024-03-17T23:57:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7feec420b740 1 -- 172.21.15.12:0/3680302504 wait complete. 2024-03-17T23:57:09.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.567+0000 7feec420b740 20 client.25238 trim_cache size 0 max 16384 2024-03-17T23:57:09.572 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-17T23:57:09.573 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:57:09.573 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-17T23:57:09.573 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24587, 2024-03-17T23:57:09.573 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-17T23:57:09.573 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-17T23:57:09.573 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-17T23:57:09.573 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 132.257282987, 2024-03-17T23:57:09.573 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 28, 2024-03-17T23:57:09.573 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 94, 2024-03-17T23:57:09.573 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 90, 2024-03-17T23:57:09.573 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 139.27392026199999 2024-03-17T23:57:09.573 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-17T23:57:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.568+0000 7feec0a00640 1 -- 172.21.15.12:0/220178251 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7feea00a2880 msgr2=0x7feea00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.568+0000 7feec0a00640 1 --2- 172.21.15.12:0/220178251 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7feea00a2880 0x7feea00a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55a30915e590 tx=0x7feeb4009500 comp rx=0 tx=0).stop 2024-03-17T23:57:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.568+0000 7feec0a00640 1 -- 172.21.15.12:0/220178251 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feebc104ad0 msgr2=0x7feebc19a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.568+0000 7feec0a00640 1 --2- 172.21.15.12:0/220178251 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feebc104ad0 0x7feebc19a1c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7feea4009230 tx=0x7feea402c670 comp rx=0 tx=0).stop 2024-03-17T23:57:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.568+0000 7feec0a00640 1 -- 172.21.15.12:0/220178251 shutdown_connections 2024-03-17T23:57:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.568+0000 7feec0a00640 1 --2- 172.21.15.12:0/220178251 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7feea00a2880 0x7feea00a4d40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.568+0000 7feec0a00640 1 --2- 172.21.15.12:0/220178251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feebc105490 0x7feebc1a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.568+0000 7feec0a00640 1 --2- 172.21.15.12:0/220178251 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feebc104ad0 0x7feebc19a1c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.568+0000 7feec0a00640 1 --2- 172.21.15.12:0/220178251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feebc1038e0 0x7feebc199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:09.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.568+0000 7feec0a00640 1 -- 172.21.15.12:0/220178251 >> 172.21.15.12:0/220178251 conn(0x7feebc0fee50 msgr2=0x7feebc107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:09.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.568+0000 7feec0a00640 1 -- 172.21.15.12:0/220178251 shutdown_connections 2024-03-17T23:57:09.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:09.568+0000 7feec0a00640 1 -- 172.21.15.12:0/220178251 wait complete. 2024-03-17T23:57:09.587 INFO:tasks.cephadm:Setup complete, yielding 2024-03-17T23:57:09.588 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-17T23:57:09.599 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-03-17T23:57:09.599 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-17T23:57:09.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:09 smithi012 ceph-mon[31541]: pgmap v391: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-17T23:57:10.062 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:57:10.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:09 smithi191 ceph-mon[36708]: pgmap v391: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-17T23:57:10.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:09 smithi079 ceph-mon[36241]: pgmap v391: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-17T23:57:12.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:11 smithi191 ceph-mon[36708]: pgmap v392: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:12.100 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:11 smithi012 ceph-mon[31541]: pgmap v392: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:12.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:11 smithi079 ceph-mon[36241]: pgmap v392: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.429+0000 7f0f3d600700 1 -- 172.21.15.12:0/2384427769 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f38102d90 msgr2=0x7f0f38105180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.429+0000 7f0f3d600700 1 --2- 172.21.15.12:0/2384427769 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f38102d90 0x7f0f38105180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009990 tx=0x7f0f28009ca0 comp rx=0 tx=0).stop 2024-03-17T23:57:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.429+0000 7f0f3d600700 1 -- 172.21.15.12:0/2384427769 shutdown_connections 2024-03-17T23:57:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.429+0000 7f0f3d600700 1 --2- 172.21.15.12:0/2384427769 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38107ff0 0x7f0f3810a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.429+0000 7f0f3d600700 1 --2- 172.21.15.12:0/2384427769 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f38102d90 0x7f0f38105180 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.429+0000 7f0f3d600700 1 --2- 172.21.15.12:0/2384427769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f381056c0 0x7f0f38107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.429+0000 7f0f3d600700 1 -- 172.21.15.12:0/2384427769 >> 172.21.15.12:0/2384427769 conn(0x7f0f380fc7a0 msgr2=0x7f0f380febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.429+0000 7f0f3d600700 1 -- 172.21.15.12:0/2384427769 shutdown_connections 2024-03-17T23:57:12.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.429+0000 7f0f3d600700 1 -- 172.21.15.12:0/2384427769 wait complete. 2024-03-17T23:57:12.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.431+0000 7f0f3d600700 1 Processor -- start 2024-03-17T23:57:12.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.431+0000 7f0f3d600700 1 -- start start 2024-03-17T23:57:12.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.431+0000 7f0f3d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f38102d90 0x7f0f3819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:12.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f3d600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381056c0 0x7f0f3819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:12.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f3d600700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38107ff0 0x7f0f381a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:12.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f3d600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3810df20 con 0x7f0f38102d90 2024-03-17T23:57:12.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f3d600700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3810dda0 con 0x7f0f38107ff0 2024-03-17T23:57:12.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f3d600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3810e0a0 con 0x7f0f381056c0 2024-03-17T23:57:12.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f36a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f38102d90 0x7f0f3819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:12.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f36000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381056c0 0x7f0f3819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:12.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f36000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381056c0 0x7f0f3819ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:53754/0 (socket says 172.21.15.12:53754) 2024-03-17T23:57:12.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f36a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f38102d90 0x7f0f3819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47262/0 (socket says 172.21.15.12:47262) 2024-03-17T23:57:12.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f36000700 1 -- 172.21.15.12:0/3951908219 learned_addr learned my addr 172.21.15.12:0/3951908219 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:12.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f37400700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38107ff0 0x7f0f381a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f36000700 1 -- 172.21.15.12:0/3951908219 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38107ff0 msgr2=0x7f0f381a1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f36000700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38107ff0 0x7f0f381a1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f36000700 1 -- 172.21.15.12:0/3951908219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f38102d90 msgr2=0x7f0f3819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f36000700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f38102d90 0x7f0f3819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f36000700 1 -- 172.21.15.12:0/3951908219 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f381a2440 con 0x7f0f381056c0 2024-03-17T23:57:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.432+0000 7f0f37400700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38107ff0 0x7f0f381a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.433+0000 7f0f36000700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381056c0 0x7f0f3819ad20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400e9d0 tx=0x7f0f2400ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.433+0000 7f0f2f400700 1 -- 172.21.15.12:0/3951908219 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2400cbd0 con 0x7f0f381056c0 2024-03-17T23:57:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.433+0000 7f0f2f400700 1 -- 172.21.15.12:0/3951908219 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f2400ee70 con 0x7f0f381056c0 2024-03-17T23:57:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.433+0000 7f0f2f400700 1 -- 172.21.15.12:0/3951908219 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24018660 con 0x7f0f381056c0 2024-03-17T23:57:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.433+0000 7f0f3d600700 1 -- 172.21.15.12:0/3951908219 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f381a2730 con 0x7f0f381056c0 2024-03-17T23:57:12.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.433+0000 7f0f3d600700 1 -- 172.21.15.12:0/3951908219 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f381a2c20 con 0x7f0f381056c0 2024-03-17T23:57:12.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.434+0000 7f0f3d600700 1 -- 172.21.15.12:0/3951908219 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f38068f90 con 0x7f0f381056c0 2024-03-17T23:57:12.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.437+0000 7f0f2f400700 1 -- 172.21.15.12:0/3951908219 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0f24010430 con 0x7f0f381056c0 2024-03-17T23:57:12.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.438+0000 7f0f2f400700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0f1c0a2910 0x7f0f1c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:12.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.438+0000 7f0f2f400700 1 -- 172.21.15.12:0/3951908219 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f0f240cf1e0 con 0x7f0f381056c0 2024-03-17T23:57:12.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.438+0000 7f0f36a00700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0f1c0a2910 0x7f0f1c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:12.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.439+0000 7f0f36a00700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0f1c0a2910 0x7f0f1c0a4dd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800b7b0 tx=0x7f0f2800b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:12.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.440+0000 7f0f2f400700 1 -- 172.21.15.12:0/3951908219 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f2409bea0 con 0x7f0f381056c0 2024-03-17T23:57:12.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.685+0000 7f0f3d600700 1 -- 172.21.15.12:0/3951908219 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f0f38062710 con 0x7f0f1c0a2910 2024-03-17T23:57:12.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.687+0000 7f0f2f400700 1 -- 172.21.15.12:0/3951908219 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f0f38062710 con 0x7f0f1c0a2910 2024-03-17T23:57:12.688 INFO:teuthology.orchestra.run.smithi012.stdout:Backend: cephadm 2024-03-17T23:57:12.688 INFO:teuthology.orchestra.run.smithi012.stdout:Available: Yes 2024-03-17T23:57:12.688 INFO:teuthology.orchestra.run.smithi012.stdout:Paused: No 2024-03-17T23:57:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.691+0000 7f0f3d600700 1 -- 172.21.15.12:0/3951908219 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0f1c0a2910 msgr2=0x7f0f1c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.691+0000 7f0f3d600700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0f1c0a2910 0x7f0f1c0a4dd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800b7b0 tx=0x7f0f2800b880 comp rx=0 tx=0).stop 2024-03-17T23:57:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.691+0000 7f0f3d600700 1 -- 172.21.15.12:0/3951908219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381056c0 msgr2=0x7f0f3819ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.691+0000 7f0f3d600700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381056c0 0x7f0f3819ad20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400e9d0 tx=0x7f0f2400ece0 comp rx=0 tx=0).stop 2024-03-17T23:57:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.692+0000 7f0f3d600700 1 -- 172.21.15.12:0/3951908219 shutdown_connections 2024-03-17T23:57:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.692+0000 7f0f3d600700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f0f1c0a2910 0x7f0f1c0a4dd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.692+0000 7f0f3d600700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0f38107ff0 0x7f0f381a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.692+0000 7f0f3d600700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f38102d90 0x7f0f3819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.692+0000 7f0f3d600700 1 --2- 172.21.15.12:0/3951908219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f381056c0 0x7f0f3819ad20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.692+0000 7f0f3d600700 1 -- 172.21.15.12:0/3951908219 >> 172.21.15.12:0/3951908219 conn(0x7f0f380fc7a0 msgr2=0x7f0f38102630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.692+0000 7f0f3d600700 1 -- 172.21.15.12:0/3951908219 shutdown_connections 2024-03-17T23:57:12.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:12.692+0000 7f0f3d600700 1 -- 172.21.15.12:0/3951908219 wait complete. 2024-03-17T23:57:13.507 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-17T23:57:13.970 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:57:14.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:13 smithi191 ceph-mon[36708]: pgmap v393: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:14.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:13 smithi191 ceph-mon[36708]: from='client.25225 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:14.125 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:13 smithi012 ceph-mon[31541]: pgmap v393: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:14.125 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:13 smithi012 ceph-mon[31541]: from='client.25225 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:14.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:13 smithi079 ceph-mon[36241]: pgmap v393: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:14.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:13 smithi079 ceph-mon[36241]: from='client.25225 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:15.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:15 smithi012 ceph-mon[31541]: pgmap v394: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:16.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:15 smithi191 ceph-mon[36708]: pgmap v394: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:16.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.178+0000 7f93c8400700 1 -- 172.21.15.12:0/145210909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93c0107ff0 msgr2=0x7f93c010a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:16.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.178+0000 7f93c8400700 1 --2- 172.21.15.12:0/145210909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93c0107ff0 0x7f93c010a460 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f93b800b0a0 tx=0x7f93b800b3b0 comp rx=0 tx=0).stop 2024-03-17T23:57:16.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.179+0000 7f93c8400700 1 -- 172.21.15.12:0/145210909 shutdown_connections 2024-03-17T23:57:16.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.179+0000 7f93c8400700 1 --2- 172.21.15.12:0/145210909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93c0107ff0 0x7f93c010a460 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:16.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.179+0000 7f93c8400700 1 --2- 172.21.15.12:0/145210909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c0102d90 0x7f93c0105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:16.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.179+0000 7f93c8400700 1 --2- 172.21.15.12:0/145210909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93c01056c0 0x7f93c0107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:16.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.179+0000 7f93c8400700 1 -- 172.21.15.12:0/145210909 >> 172.21.15.12:0/145210909 conn(0x7f93c00fc7a0 msgr2=0x7f93c00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:16.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.179+0000 7f93c8400700 1 -- 172.21.15.12:0/145210909 shutdown_connections 2024-03-17T23:57:16.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.179+0000 7f93c8400700 1 -- 172.21.15.12:0/145210909 wait complete. 2024-03-17T23:57:16.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.180+0000 7f93c8400700 1 Processor -- start 2024-03-17T23:57:16.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.180+0000 7f93c8400700 1 -- start start 2024-03-17T23:57:16.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.180+0000 7f93c8400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c0102d90 0x7f93c01026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:16.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.180+0000 7f93c8400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93c01056c0 0x7f93c0100d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:16.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c8400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93c0107ff0 0x7f93c0101280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:16.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c8400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f93c010df60 con 0x7f93c0107ff0 2024-03-17T23:57:16.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c8400700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f93c010dde0 con 0x7f93c0102d90 2024-03-17T23:57:16.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c8400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f93c010e0e0 con 0x7f93c01056c0 2024-03-17T23:57:16.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c5a00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c0102d90 0x7f93c01026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:16.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c5a00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c0102d90 0x7f93c01026f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56906/0 (socket says 172.21.15.12:56906) 2024-03-17T23:57:16.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c5a00700 1 -- 172.21.15.12:0/1360294486 learned_addr learned my addr 172.21.15.12:0/1360294486 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:16.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c5000700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93c01056c0 0x7f93c0100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:16.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c6400700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93c0107ff0 0x7f93c0101280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:16.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c5a00700 1 -- 172.21.15.12:0/1360294486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93c01056c0 msgr2=0x7f93c0100d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:16.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c5a00700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93c01056c0 0x7f93c0100d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:16.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c5a00700 1 -- 172.21.15.12:0/1360294486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93c0107ff0 msgr2=0x7f93c0101280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:16.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c5a00700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93c0107ff0 0x7f93c0101280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:16.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c5a00700 1 -- 172.21.15.12:0/1360294486 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c0101b30 con 0x7f93c0102d90 2024-03-17T23:57:16.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.181+0000 7f93c5000700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93c01056c0 0x7f93c0100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:16.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.182+0000 7f93c5a00700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c0102d90 0x7f93c01026f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f93b400e9c0 tx=0x7f93b400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:16.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.182+0000 7f93be000700 1 -- 172.21.15.12:0/1360294486 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b400cbd0 con 0x7f93c0102d90 2024-03-17T23:57:16.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.182+0000 7f93be000700 1 -- 172.21.15.12:0/1360294486 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93b400cd70 con 0x7f93c0102d90 2024-03-17T23:57:16.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.182+0000 7f93be000700 1 -- 172.21.15.12:0/1360294486 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b4010430 con 0x7f93c0102d90 2024-03-17T23:57:16.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.182+0000 7f93c8400700 1 -- 172.21.15.12:0/1360294486 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c01a26d0 con 0x7f93c0102d90 2024-03-17T23:57:16.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.182+0000 7f93c8400700 1 -- 172.21.15.12:0/1360294486 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c01a2be0 con 0x7f93c0102d90 2024-03-17T23:57:16.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.183+0000 7f93c8400700 1 -- 172.21.15.12:0/1360294486 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93c0068f90 con 0x7f93c0102d90 2024-03-17T23:57:16.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.187+0000 7f93be000700 1 -- 172.21.15.12:0/1360294486 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f93b4004750 con 0x7f93c0102d90 2024-03-17T23:57:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.188+0000 7f93be000700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f93a80a2910 0x7f93a80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.188+0000 7f93be000700 1 -- 172.21.15.12:0/1360294486 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f93b40cf080 con 0x7f93c0102d90 2024-03-17T23:57:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.188+0000 7f93c5000700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f93a80a2910 0x7f93a80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:16.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.189+0000 7f93c5000700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f93a80a2910 0x7f93a80a4dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f93ac011490 tx=0x7f93ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:16.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.192+0000 7f93be000700 1 -- 172.21.15.12:0/1360294486 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f93b409bd40 con 0x7f93c0102d90 2024-03-17T23:57:16.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:15 smithi079 ceph-mon[36241]: pgmap v394: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:16.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.436+0000 7f93c8400700 1 -- 172.21.15.12:0/1360294486 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f93c0062710 con 0x7f93a80a2910 2024-03-17T23:57:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.460+0000 7f93be000700 1 -- 172.21.15.12:0/1360294486 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f93c0062710 con 0x7f93a80a2910 2024-03-17T23:57:16.461 INFO:teuthology.orchestra.run.smithi012.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-17T23:57:16.461 INFO:teuthology.orchestra.run.smithi012.stdout:mds.a smithi012 running (2m) 29s ago 2m 16.8M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 1d70a05843c8 2024-03-17T23:57:16.461 INFO:teuthology.orchestra.run.smithi012.stdout:mds.b smithi079 running (2m) 77s ago 2m 25.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 90b5110dc80e 2024-03-17T23:57:16.461 INFO:teuthology.orchestra.run.smithi012.stdout:mds.c smithi191 running (2m) 76s ago 2m 17.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 949e250428f2 2024-03-17T23:57:16.462 INFO:teuthology.orchestra.run.smithi012.stdout:mds.d smithi012 running (2m) 29s ago 2m 15.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b d0bed2ffb65e 2024-03-17T23:57:16.462 INFO:teuthology.orchestra.run.smithi012.stdout:mds.e smithi079 running (2m) 77s ago 2m 16.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b c3b0811d0d33 2024-03-17T23:57:16.462 INFO:teuthology.orchestra.run.smithi012.stdout:mds.f smithi191 running (2m) 76s ago 2m 18.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b fad973357eed 2024-03-17T23:57:16.462 INFO:teuthology.orchestra.run.smithi012.stdout:mds.g smithi012 running (2m) 29s ago 2m 12.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 2c8d7200ec4c 2024-03-17T23:57:16.462 INFO:teuthology.orchestra.run.smithi012.stdout:mds.h smithi079 running (2m) 77s ago 2m 17.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 7386f8afc90f 2024-03-17T23:57:16.462 INFO:teuthology.orchestra.run.smithi012.stdout:mds.i smithi191 running (2m) 76s ago 2m 16.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 8439bf529508 2024-03-17T23:57:16.462 INFO:teuthology.orchestra.run.smithi012.stdout:mds.j smithi012 running (2m) 29s ago 2m 12.5M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 07026a04c1a6 2024-03-17T23:57:16.462 INFO:teuthology.orchestra.run.smithi012.stdout:mds.k smithi079 running (3m) 77s ago 3m 15.9M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 08729b28bcff 2024-03-17T23:57:16.462 INFO:teuthology.orchestra.run.smithi012.stdout:mds.l smithi191 running (3m) 76s ago 3m 17.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b b100e76a6ff3 2024-03-17T23:57:16.462 INFO:teuthology.orchestra.run.smithi012.stdout:mgr.x smithi012 *:9283,8765 running (11m) 29s ago 11m 514M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 55a544c384fb 2024-03-17T23:57:16.462 INFO:teuthology.orchestra.run.smithi012.stdout:mgr.y smithi079 *:8443,8765 running (8m) 77s ago 8m 453M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 48dbac523852 2024-03-17T23:57:16.462 INFO:teuthology.orchestra.run.smithi012.stdout:mgr.z smithi191 *:8443,8765 running (9m) 76s ago 9m 454M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 436c47a81ab9 2024-03-17T23:57:16.462 INFO:teuthology.orchestra.run.smithi012.stdout:mon.a smithi012 running (11m) 29s ago 11m 47.9M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b ca282921d134 2024-03-17T23:57:16.463 INFO:teuthology.orchestra.run.smithi012.stdout:mon.b smithi079 running (9m) 77s ago 9m 45.0M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 3d8775146265 2024-03-17T23:57:16.463 INFO:teuthology.orchestra.run.smithi012.stdout:mon.c smithi191 running (9m) 76s ago 9m 41.3M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 90dcfc59bc5d 2024-03-17T23:57:16.463 INFO:teuthology.orchestra.run.smithi012.stdout:osd.0 smithi012 running (8m) 29s ago 8m 55.4M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b d7b054394554 2024-03-17T23:57:16.463 INFO:teuthology.orchestra.run.smithi012.stdout:osd.1 smithi079 running (8m) 77s ago 8m 47.1M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 4ba9a1c4a46b 2024-03-17T23:57:16.463 INFO:teuthology.orchestra.run.smithi012.stdout:osd.2 smithi191 running (7m) 76s ago 7m 50.0M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b e5b0aed659c1 2024-03-17T23:57:16.463 INFO:teuthology.orchestra.run.smithi012.stdout:osd.3 smithi012 running (7m) 29s ago 7m 54.8M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b abe7553e06dc 2024-03-17T23:57:16.463 INFO:teuthology.orchestra.run.smithi012.stdout:osd.4 smithi079 running (6m) 77s ago 6m 53.4M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b b2a9bcb71108 2024-03-17T23:57:16.463 INFO:teuthology.orchestra.run.smithi012.stdout:osd.5 smithi191 running (6m) 76s ago 6m 45.5M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 4212a71e1fcd 2024-03-17T23:57:16.463 INFO:teuthology.orchestra.run.smithi012.stdout:osd.6 smithi012 running (5m) 29s ago 5m 40.7M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b 101179b74bdd 2024-03-17T23:57:16.463 INFO:teuthology.orchestra.run.smithi012.stdout:osd.7 smithi079 running (5m) 77s ago 5m 47.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 17deea3df589 2024-03-17T23:57:16.463 INFO:teuthology.orchestra.run.smithi012.stdout:osd.8 smithi191 running (4m) 76s ago 4m 46.0M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b c0e51772638b 2024-03-17T23:57:16.463 INFO:teuthology.orchestra.run.smithi012.stdout:osd.9 smithi012 running (4m) 29s ago 4m 52.2M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b 0c874e1ddc05 2024-03-17T23:57:16.463 INFO:teuthology.orchestra.run.smithi012.stdout:osd.10 smithi079 running (3m) 77s ago 3m 48.1M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b b4c4727397e9 2024-03-17T23:57:16.464 INFO:teuthology.orchestra.run.smithi012.stdout:osd.11 smithi191 running (3m) 76s ago 3m 41.7M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 369608f8aa62 2024-03-17T23:57:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.462+0000 7f93c8400700 1 -- 172.21.15.12:0/1360294486 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f93a80a2910 msgr2=0x7f93a80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.462+0000 7f93c8400700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f93a80a2910 0x7f93a80a4dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f93ac011490 tx=0x7f93ac009260 comp rx=0 tx=0).stop 2024-03-17T23:57:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.462+0000 7f93c8400700 1 -- 172.21.15.12:0/1360294486 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c0102d90 msgr2=0x7f93c01026f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.462+0000 7f93c8400700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c0102d90 0x7f93c01026f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f93b400e9c0 tx=0x7f93b400ecd0 comp rx=0 tx=0).stop 2024-03-17T23:57:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.463+0000 7f93c8400700 1 -- 172.21.15.12:0/1360294486 shutdown_connections 2024-03-17T23:57:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.463+0000 7f93c8400700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f93a80a2910 0x7f93a80a4dd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.463+0000 7f93c8400700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93c0107ff0 0x7f93c0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.463+0000 7f93c8400700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c0102d90 0x7f93c01026f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.463+0000 7f93c8400700 1 --2- 172.21.15.12:0/1360294486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93c01056c0 0x7f93c0100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.463+0000 7f93c8400700 1 -- 172.21.15.12:0/1360294486 >> 172.21.15.12:0/1360294486 conn(0x7f93c00fc7a0 msgr2=0x7f93c01036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.463+0000 7f93c8400700 1 -- 172.21.15.12:0/1360294486 shutdown_connections 2024-03-17T23:57:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:16.463+0000 7f93c8400700 1 -- 172.21.15.12:0/1360294486 wait complete. 2024-03-17T23:57:17.333 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-17T23:57:17.811 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:57:18.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:17 smithi191 ceph-mon[36708]: pgmap v395: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:57:18.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:17 smithi191 ceph-mon[36708]: from='client.25250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:18.096 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:17 smithi079 ceph-mon[36241]: pgmap v395: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:57:18.096 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:17 smithi079 ceph-mon[36241]: from='client.25250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:18.104 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:17 smithi012 ceph-mon[31541]: pgmap v395: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:57:18.104 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:17 smithi012 ceph-mon[31541]: from='client.25250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:20.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.032+0000 7f1129400700 1 -- 172.21.15.12:0/3666079538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1124107ff0 msgr2=0x7f112410a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:20.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.032+0000 7f1129400700 1 --2- 172.21.15.12:0/3666079538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1124107ff0 0x7f112410a460 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f111c00b0a0 tx=0x7f111c00b3b0 comp rx=0 tx=0).stop 2024-03-17T23:57:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.032+0000 7f1129400700 1 -- 172.21.15.12:0/3666079538 shutdown_connections 2024-03-17T23:57:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.032+0000 7f1129400700 1 --2- 172.21.15.12:0/3666079538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1124107ff0 0x7f112410a460 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.032+0000 7f1129400700 1 --2- 172.21.15.12:0/3666079538 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124102d90 0x7f1124105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.032+0000 7f1129400700 1 --2- 172.21.15.12:0/3666079538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11241056c0 0x7f1124107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.032+0000 7f1129400700 1 -- 172.21.15.12:0/3666079538 >> 172.21.15.12:0/3666079538 conn(0x7f11240fc7a0 msgr2=0x7f11240febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.032+0000 7f1129400700 1 -- 172.21.15.12:0/3666079538 shutdown_connections 2024-03-17T23:57:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.033+0000 7f1129400700 1 -- 172.21.15.12:0/3666079538 wait complete. 2024-03-17T23:57:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.034+0000 7f1129400700 1 Processor -- start 2024-03-17T23:57:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.034+0000 7f1129400700 1 -- start start 2024-03-17T23:57:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.034+0000 7f1129400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124102d90 0x7f112419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.034+0000 7f1129400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11241056c0 0x7f112419ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.034+0000 7f1129400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1124107ff0 0x7f11241a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.034+0000 7f1129400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f112410df60 con 0x7f11241056c0 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.034+0000 7f1129400700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f112410dde0 con 0x7f1124102d90 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.034+0000 7f1129400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f112410e0e0 con 0x7f1124107ff0 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.034+0000 7f1122800700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124102d90 0x7f112419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.034+0000 7f1119e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11241056c0 0x7f112419ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.034+0000 7f1122800700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124102d90 0x7f112419a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:56924/0 (socket says 172.21.15.12:56924) 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.034+0000 7f1122800700 1 -- 172.21.15.12:0/2558619427 learned_addr learned my addr 172.21.15.12:0/2558619427 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.035+0000 7f1123200700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1124107ff0 0x7f11241a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.035+0000 7f1119e00700 1 -- 172.21.15.12:0/2558619427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1124107ff0 msgr2=0x7f11241a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.035+0000 7f1119e00700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1124107ff0 0x7f11241a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.035+0000 7f1119e00700 1 -- 172.21.15.12:0/2558619427 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124102d90 msgr2=0x7f112419a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.035+0000 7f1119e00700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124102d90 0x7f112419a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.035+0000 7f1119e00700 1 -- 172.21.15.12:0/2558619427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11241a23a0 con 0x7f11241056c0 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.035+0000 7f1122800700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124102d90 0x7f112419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.035+0000 7f1119e00700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11241056c0 0x7f112419ac80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f110c00ea20 tx=0x7f110c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.036+0000 7f111be00700 1 -- 172.21.15.12:0/2558619427 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c00cbd0 con 0x7f11241056c0 2024-03-17T23:57:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.036+0000 7f111be00700 1 -- 172.21.15.12:0/2558619427 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f110c00ee60 con 0x7f11241056c0 2024-03-17T23:57:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.036+0000 7f1129400700 1 -- 172.21.15.12:0/2558619427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11241a2690 con 0x7f11241056c0 2024-03-17T23:57:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.036+0000 7f111be00700 1 -- 172.21.15.12:0/2558619427 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c018680 con 0x7f11241056c0 2024-03-17T23:57:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.036+0000 7f1129400700 1 -- 172.21.15.12:0/2558619427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11241a2bd0 con 0x7f11241056c0 2024-03-17T23:57:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.037+0000 7f1129400700 1 -- 172.21.15.12:0/2558619427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1124068f90 con 0x7f11241056c0 2024-03-17T23:57:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.039+0000 7f111be00700 1 -- 172.21.15.12:0/2558619427 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f110c018820 con 0x7f11241056c0 2024-03-17T23:57:20.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.041+0000 7f111be00700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f11040a2890 0x7f11040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:20.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.042+0000 7f111be00700 1 -- 172.21.15.12:0/2558619427 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f110c0cef50 con 0x7f11241056c0 2024-03-17T23:57:20.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.042+0000 7f1122800700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f11040a2890 0x7f11040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:20.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.043+0000 7f1122800700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f11040a2890 0x7f11040a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1114011550 tx=0x7f1114009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.047+0000 7f111be00700 1 -- 172.21.15.12:0/2558619427 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f110c09bc10 con 0x7f11241056c0 2024-03-17T23:57:20.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:19 smithi191 ceph-mon[36708]: pgmap v396: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:20.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:19 smithi012 ceph-mon[31541]: pgmap v396: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:20.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.287+0000 7f1129400700 1 -- 172.21.15.12:0/2558619427 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f1124062710 con 0x7f11040a2890 2024-03-17T23:57:20.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:19 smithi079 ceph-mon[36241]: pgmap v396: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:20.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.299+0000 7f111be00700 1 -- 172.21.15.12:0/2558619427 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f1124062710 con 0x7f11040a2890 2024-03-17T23:57:20.300 INFO:teuthology.orchestra.run.smithi012.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-17T23:57:20.301 INFO:teuthology.orchestra.run.smithi012.stdout:mds.all 12/12 80s ago 3m smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-03-17T23:57:20.301 INFO:teuthology.orchestra.run.smithi012.stdout:mgr 3/3 80s ago 9m smithi012=x;smithi079=y;smithi191=z;count:3 2024-03-17T23:57:20.301 INFO:teuthology.orchestra.run.smithi012.stdout:mon 3/3 80s ago 9m smithi012:172.21.15.12=a;smithi079:172.21.15.79=b;smithi191:172.21.15.191=c;count:3 2024-03-17T23:57:20.301 INFO:teuthology.orchestra.run.smithi012.stdout:osd 12 80s ago - 2024-03-17T23:57:20.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.302+0000 7f1129400700 1 -- 172.21.15.12:0/2558619427 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f11040a2890 msgr2=0x7f11040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.302+0000 7f1129400700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f11040a2890 0x7f11040a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1114011550 tx=0x7f1114009260 comp rx=0 tx=0).stop 2024-03-17T23:57:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.302+0000 7f1129400700 1 -- 172.21.15.12:0/2558619427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11241056c0 msgr2=0x7f112419ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.302+0000 7f1129400700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11241056c0 0x7f112419ac80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f110c00ea20 tx=0x7f110c00ed30 comp rx=0 tx=0).stop 2024-03-17T23:57:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.303+0000 7f1129400700 1 -- 172.21.15.12:0/2558619427 shutdown_connections 2024-03-17T23:57:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.303+0000 7f1129400700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f11040a2890 0x7f11040a4d50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.303+0000 7f1129400700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1124107ff0 0x7f11241a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.303+0000 7f1129400700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1124102d90 0x7f112419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.303+0000 7f1129400700 1 --2- 172.21.15.12:0/2558619427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11241056c0 0x7f112419ac80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.303+0000 7f1129400700 1 -- 172.21.15.12:0/2558619427 >> 172.21.15.12:0/2558619427 conn(0x7f11240fc7a0 msgr2=0x7f1124102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.303+0000 7f1129400700 1 -- 172.21.15.12:0/2558619427 shutdown_connections 2024-03-17T23:57:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:20.303+0000 7f1129400700 1 -- 172.21.15.12:0/2558619427 wait complete. 2024-03-17T23:57:21.044 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-17T23:57:21.530 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:57:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:21 smithi012 ceph-mon[31541]: pgmap v397: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:22.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:21 smithi012 ceph-mon[31541]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:22.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:21 smithi191 ceph-mon[36708]: pgmap v397: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:22.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:21 smithi191 ceph-mon[36708]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:22.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:21 smithi079 ceph-mon[36241]: pgmap v397: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:22.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:21 smithi079 ceph-mon[36241]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.858+0000 7f62c0800700 1 -- 172.21.15.12:0/192816174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62b8102d90 msgr2=0x7f62b8105180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.858+0000 7f62c0800700 1 --2- 172.21.15.12:0/192816174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62b8102d90 0x7f62b8105180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f62ac009990 tx=0x7f62ac009ca0 comp rx=0 tx=0).stop 2024-03-17T23:57:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.858+0000 7f62c0800700 1 -- 172.21.15.12:0/192816174 shutdown_connections 2024-03-17T23:57:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.858+0000 7f62c0800700 1 --2- 172.21.15.12:0/192816174 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f62b8107ff0 0x7f62b810a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.858+0000 7f62c0800700 1 --2- 172.21.15.12:0/192816174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62b8102d90 0x7f62b8105180 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.858+0000 7f62c0800700 1 --2- 172.21.15.12:0/192816174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b81056c0 0x7f62b8107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.858+0000 7f62c0800700 1 -- 172.21.15.12:0/192816174 >> 172.21.15.12:0/192816174 conn(0x7f62b80fc7a0 msgr2=0x7f62b80febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.859+0000 7f62c0800700 1 -- 172.21.15.12:0/192816174 shutdown_connections 2024-03-17T23:57:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.859+0000 7f62c0800700 1 -- 172.21.15.12:0/192816174 wait complete. 2024-03-17T23:57:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.860+0000 7f62c0800700 1 Processor -- start 2024-03-17T23:57:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.861+0000 7f62c0800700 1 -- start start 2024-03-17T23:57:23.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.861+0000 7f62c0800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b8102d90 0x7f62b81026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:23.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.861+0000 7f62c0800700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f62b81056c0 0x7f62b8100d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:23.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.861+0000 7f62c0800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62b8107ff0 0x7f62b8101280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:23.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.861+0000 7f62c0800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f62b810df10 con 0x7f62b8102d90 2024-03-17T23:57:23.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.861+0000 7f62c0800700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f62b810dd90 con 0x7f62b81056c0 2024-03-17T23:57:23.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.861+0000 7f62c0800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f62b810e090 con 0x7f62b8107ff0 2024-03-17T23:57:23.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.861+0000 7f62bde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b8102d90 0x7f62b81026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:23.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.861+0000 7f62bd400700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f62b81056c0 0x7f62b8100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:23.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.861+0000 7f62bde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b8102d90 0x7f62b81026f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59466/0 (socket says 172.21.15.12:59466) 2024-03-17T23:57:23.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.861+0000 7f62bde00700 1 -- 172.21.15.12:0/1602133745 learned_addr learned my addr 172.21.15.12:0/1602133745 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:23.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.862+0000 7f62be800700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62b8107ff0 0x7f62b8101280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:23.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.862+0000 7f62bd400700 1 -- 172.21.15.12:0/1602133745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62b8107ff0 msgr2=0x7f62b8101280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:23.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.862+0000 7f62bd400700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62b8107ff0 0x7f62b8101280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:23.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.862+0000 7f62bd400700 1 -- 172.21.15.12:0/1602133745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b8102d90 msgr2=0x7f62b81026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:23.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.862+0000 7f62bd400700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b8102d90 0x7f62b81026f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:23.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.862+0000 7f62bd400700 1 -- 172.21.15.12:0/1602133745 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b8101b30 con 0x7f62b81056c0 2024-03-17T23:57:23.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.862+0000 7f62be800700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62b8107ff0 0x7f62b8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:23.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.862+0000 7f62bde00700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b8102d90 0x7f62b81026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.862+0000 7f62bd400700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f62b81056c0 0x7f62b8100d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f62a800c970 tx=0x7f62a800cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.863+0000 7f62b2a00700 1 -- 172.21.15.12:0/1602133745 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8014070 con 0x7f62b81056c0 2024-03-17T23:57:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.863+0000 7f62b2a00700 1 -- 172.21.15.12:0/1602133745 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62a8004440 con 0x7f62b81056c0 2024-03-17T23:57:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.863+0000 7f62c0800700 1 -- 172.21.15.12:0/1602133745 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b81a2660 con 0x7f62b81056c0 2024-03-17T23:57:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.863+0000 7f62b2a00700 1 -- 172.21.15.12:0/1602133745 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8007b60 con 0x7f62b81056c0 2024-03-17T23:57:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.863+0000 7f62c0800700 1 -- 172.21.15.12:0/1602133745 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b81a2ba0 con 0x7f62b81056c0 2024-03-17T23:57:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.864+0000 7f62c0800700 1 -- 172.21.15.12:0/1602133745 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62b8068f90 con 0x7f62b81056c0 2024-03-17T23:57:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.866+0000 7f62b2a00700 1 -- 172.21.15.12:0/1602133745 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f62a8017070 con 0x7f62b81056c0 2024-03-17T23:57:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.868+0000 7f62b2a00700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f62a00a2910 0x7f62a00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.868+0000 7f62b2a00700 1 -- 172.21.15.12:0/1602133745 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f62a80ce9c0 con 0x7f62b81056c0 2024-03-17T23:57:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.868+0000 7f62bde00700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f62a00a2910 0x7f62a00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.869+0000 7f62bde00700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f62a00a2910 0x7f62a00a4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00b7b0 tx=0x7f62ac00b8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:23.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:23.872+0000 7f62b2a00700 1 -- 172.21.15.12:0/1602133745 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62a801e020 con 0x7f62b81056c0 2024-03-17T23:57:24.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:23 smithi191 ceph-mon[36708]: pgmap v398: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:24.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.141+0000 7f62c0800700 1 -- 172.21.15.12:0/1602133745 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f62b8062710 con 0x7f62a00a2910 2024-03-17T23:57:24.143 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:23 smithi012 ceph-mon[31541]: pgmap v398: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:24.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.144+0000 7f62b2a00700 1 -- 172.21.15.12:0/1602133745 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f62b8062710 con 0x7f62a00a2910 2024-03-17T23:57:24.145 INFO:teuthology.orchestra.run.smithi012.stdout:HOST ADDR LABELS STATUS 2024-03-17T23:57:24.145 INFO:teuthology.orchestra.run.smithi012.stdout:smithi012 172.21.15.12 2024-03-17T23:57:24.145 INFO:teuthology.orchestra.run.smithi012.stdout:smithi079 172.21.15.79 2024-03-17T23:57:24.145 INFO:teuthology.orchestra.run.smithi012.stdout:smithi191 172.21.15.191 2024-03-17T23:57:24.146 INFO:teuthology.orchestra.run.smithi012.stdout:3 hosts in cluster 2024-03-17T23:57:24.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.148+0000 7f62c0800700 1 -- 172.21.15.12:0/1602133745 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f62a00a2910 msgr2=0x7f62a00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:24.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.148+0000 7f62c0800700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f62a00a2910 0x7f62a00a4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00b7b0 tx=0x7f62ac00b8a0 comp rx=0 tx=0).stop 2024-03-17T23:57:24.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.148+0000 7f62c0800700 1 -- 172.21.15.12:0/1602133745 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f62b81056c0 msgr2=0x7f62b8100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.148+0000 7f62c0800700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f62b81056c0 0x7f62b8100d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f62a800c970 tx=0x7f62a800cc80 comp rx=0 tx=0).stop 2024-03-17T23:57:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.149+0000 7f62c0800700 1 -- 172.21.15.12:0/1602133745 shutdown_connections 2024-03-17T23:57:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.149+0000 7f62c0800700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f62a00a2910 0x7f62a00a4dd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.149+0000 7f62c0800700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62b8107ff0 0x7f62b8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.149+0000 7f62c0800700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62b8102d90 0x7f62b81026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.149+0000 7f62c0800700 1 --2- 172.21.15.12:0/1602133745 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f62b81056c0 0x7f62b8100d40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.149+0000 7f62c0800700 1 -- 172.21.15.12:0/1602133745 >> 172.21.15.12:0/1602133745 conn(0x7f62b80fc7a0 msgr2=0x7f62b8103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.149+0000 7f62c0800700 1 -- 172.21.15.12:0/1602133745 shutdown_connections 2024-03-17T23:57:24.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:24.149+0000 7f62c0800700 1 -- 172.21.15.12:0/1602133745 wait complete. 2024-03-17T23:57:24.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:23 smithi079 ceph-mon[36241]: pgmap v398: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:25.018 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-17T23:57:25.502 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:57:25.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:25 smithi012 ceph-mon[31541]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:25.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:25 smithi012 ceph-mon[31541]: pgmap v399: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:26.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:25 smithi191 ceph-mon[36708]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:26.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:25 smithi191 ceph-mon[36708]: pgmap v399: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:26.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:25 smithi079 ceph-mon[36241]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:26.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:25 smithi079 ceph-mon[36241]: pgmap v399: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:27.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.792+0000 7fe6aec00700 1 -- 172.21.15.12:0/243798815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6a8103e70 msgr2=0x7fe6a810a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:27.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.792+0000 7fe6aec00700 1 --2- 172.21.15.12:0/243798815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6a8103e70 0x7fe6a810a720 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000b0a0 tx=0x7fe6a000b3b0 comp rx=0 tx=0).stop 2024-03-17T23:57:27.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.792+0000 7fe6aec00700 1 -- 172.21.15.12:0/243798815 shutdown_connections 2024-03-17T23:57:27.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.792+0000 7fe6aec00700 1 --2- 172.21.15.12:0/243798815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6a8103e70 0x7fe6a810a720 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:27.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.792+0000 7fe6aec00700 1 --2- 172.21.15.12:0/243798815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6a81022d0 0x7fe6a81026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:27.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.792+0000 7fe6aec00700 1 --2- 172.21.15.12:0/243798815 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6a81034d0 0x7fe6a8103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:27.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.792+0000 7fe6aec00700 1 -- 172.21.15.12:0/243798815 >> 172.21.15.12:0/243798815 conn(0x7fe6a80fd8b0 msgr2=0x7fe6a80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:27.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.793+0000 7fe6aec00700 1 -- 172.21.15.12:0/243798815 shutdown_connections 2024-03-17T23:57:27.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.793+0000 7fe6aec00700 1 -- 172.21.15.12:0/243798815 wait complete. 2024-03-17T23:57:27.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.794+0000 7fe6aec00700 1 Processor -- start 2024-03-17T23:57:27.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.794+0000 7fe6aec00700 1 -- start start 2024-03-17T23:57:27.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.795+0000 7fe6aec00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6a81022d0 0x7fe6a819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:27.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.795+0000 7fe6aec00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6a81034d0 0x7fe6a819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:27.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.795+0000 7fe6aec00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6a8103e70 0x7fe6a81a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:27.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.795+0000 7fe6aec00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a810def0 con 0x7fe6a81022d0 2024-03-17T23:57:27.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.795+0000 7fe6aec00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a810dd70 con 0x7fe6a81034d0 2024-03-17T23:57:27.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.795+0000 7fe6aec00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a810e070 con 0x7fe6a8103e70 2024-03-17T23:57:27.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.795+0000 7fe6a7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6a81022d0 0x7fe6a819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:27.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.795+0000 7fe69fe00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6a81034d0 0x7fe6a819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:27.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.795+0000 7fe69fe00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6a81034d0 0x7fe6a819ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:58410/0 (socket says 172.21.15.12:58410) 2024-03-17T23:57:27.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.795+0000 7fe69fe00700 1 -- 172.21.15.12:0/3490112695 learned_addr learned my addr 172.21.15.12:0/3490112695 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:27.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.795+0000 7fe6acc00700 1 --2- 172.21.15.12:0/3490112695 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6a8103e70 0x7fe6a81a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:27.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.796+0000 7fe69fe00700 1 -- 172.21.15.12:0/3490112695 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6a8103e70 msgr2=0x7fe6a81a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:27.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.796+0000 7fe69fe00700 1 --2- 172.21.15.12:0/3490112695 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6a8103e70 0x7fe6a81a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:27.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.796+0000 7fe69fe00700 1 -- 172.21.15.12:0/3490112695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6a81022d0 msgr2=0x7fe6a819a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:27.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.796+0000 7fe69fe00700 1 --2- 172.21.15.12:0/3490112695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6a81022d0 0x7fe6a819a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:27.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.796+0000 7fe69fe00700 1 -- 172.21.15.12:0/3490112695 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a81a2350 con 0x7fe6a81034d0 2024-03-17T23:57:27.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.796+0000 7fe69fe00700 1 --2- 172.21.15.12:0/3490112695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6a81034d0 0x7fe6a819ac30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe69000eb30 tx=0x7fe69000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:27.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.796+0000 7fe6a5600700 1 -- 172.21.15.12:0/3490112695 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69000cbd0 con 0x7fe6a81034d0 2024-03-17T23:57:27.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.796+0000 7fe6aec00700 1 -- 172.21.15.12:0/3490112695 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6a81a2640 con 0x7fe6a81034d0 2024-03-17T23:57:27.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.796+0000 7fe6a5600700 1 -- 172.21.15.12:0/3490112695 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe690004500 con 0x7fe6a81034d0 2024-03-17T23:57:27.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.796+0000 7fe6a5600700 1 -- 172.21.15.12:0/3490112695 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe690010430 con 0x7fe6a81034d0 2024-03-17T23:57:27.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.796+0000 7fe6aec00700 1 -- 172.21.15.12:0/3490112695 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6a81a2bd0 con 0x7fe6a81034d0 2024-03-17T23:57:27.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.798+0000 7fe6aec00700 1 -- 172.21.15.12:0/3490112695 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6a8068f90 con 0x7fe6a81034d0 2024-03-17T23:57:27.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.799+0000 7fe6a5600700 1 -- 172.21.15.12:0/3490112695 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe690003f30 con 0x7fe6a81034d0 2024-03-17T23:57:27.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.801+0000 7fe6a5600700 1 --2- 172.21.15.12:0/3490112695 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe68c0a2910 0x7fe68c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:27.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.801+0000 7fe6a5600700 1 -- 172.21.15.12:0/3490112695 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fe6900cf0d0 con 0x7fe6a81034d0 2024-03-17T23:57:27.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.801+0000 7fe6a7e00700 1 --2- 172.21.15.12:0/3490112695 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe68c0a2910 0x7fe68c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:27.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.802+0000 7fe6a7e00700 1 --2- 172.21.15.12:0/3490112695 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe68c0a2910 0x7fe68c0a4dd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8103330 tx=0x7fe698009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:27.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:27.805+0000 7fe6a5600700 1 -- 172.21.15.12:0/3490112695 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe69009bd90 con 0x7fe6a81034d0 2024-03-17T23:57:27.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:27 smithi012 ceph-mon[31541]: pgmap v400: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.049+0000 7fe6aec00700 1 -- 172.21.15.12:0/3490112695 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fe6a8075c20 con 0x7fe68c0a2910 2024-03-17T23:57:28.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.051+0000 7fe6a5600700 1 -- 172.21.15.12:0/3490112695 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fe6a8075c20 con 0x7fe68c0a2910 2024-03-17T23:57:28.052 INFO:teuthology.orchestra.run.smithi012.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-17T23:57:28.052 INFO:teuthology.orchestra.run.smithi012.stdout:smithi012 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331005J400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-03-17T23:57:28.052 INFO:teuthology.orchestra.run.smithi012.stdout:smithi079 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204013G400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-17T23:57:28.052 INFO:teuthology.orchestra.run.smithi012.stdout:smithi191 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6303008M400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-17T23:57:28.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.054+0000 7fe6aec00700 1 -- 172.21.15.12:0/3490112695 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe68c0a2910 msgr2=0x7fe68c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:28.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.054+0000 7fe6aec00700 1 --2- 172.21.15.12:0/3490112695 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe68c0a2910 0x7fe68c0a4dd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8103330 tx=0x7fe698009260 comp rx=0 tx=0).stop 2024-03-17T23:57:28.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.054+0000 7fe6aec00700 1 -- 172.21.15.12:0/3490112695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6a81034d0 msgr2=0x7fe6a819ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:28.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.054+0000 7fe6aec00700 1 --2- 172.21.15.12:0/3490112695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6a81034d0 0x7fe6a819ac30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe69000eb30 tx=0x7fe69000ee40 comp rx=0 tx=0).stop 2024-03-17T23:57:28.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.055+0000 7fe6aec00700 1 -- 172.21.15.12:0/3490112695 shutdown_connections 2024-03-17T23:57:28.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.055+0000 7fe6aec00700 1 --2- 172.21.15.12:0/3490112695 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fe68c0a2910 0x7fe68c0a4dd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:28.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.055+0000 7fe6aec00700 1 --2- 172.21.15.12:0/3490112695 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6a8103e70 0x7fe6a81a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:28.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.055+0000 7fe6aec00700 1 --2- 172.21.15.12:0/3490112695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6a81022d0 0x7fe6a819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:28.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.055+0000 7fe6aec00700 1 --2- 172.21.15.12:0/3490112695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6a81034d0 0x7fe6a819ac30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:28.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.055+0000 7fe6aec00700 1 -- 172.21.15.12:0/3490112695 >> 172.21.15.12:0/3490112695 conn(0x7fe6a80fd8b0 msgr2=0x7fe6a8074500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:28.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.055+0000 7fe6aec00700 1 -- 172.21.15.12:0/3490112695 shutdown_connections 2024-03-17T23:57:28.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:28.055+0000 7fe6aec00700 1 -- 172.21.15.12:0/3490112695 wait complete. 2024-03-17T23:57:28.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:27 smithi191 ceph-mon[36708]: pgmap v400: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:57:28.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:27 smithi079 ceph-mon[36241]: pgmap v400: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:57:28.977 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-17T23:57:28.987 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-03-17T23:57:28.987 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-17T23:57:29.451 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:57:29.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:29 smithi012 ceph-mon[31541]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:29.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:29 smithi012 ceph-mon[31541]: pgmap v401: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:30.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:29 smithi191 ceph-mon[36708]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:30.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:29 smithi191 ceph-mon[36708]: pgmap v401: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:30.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:29 smithi079 ceph-mon[36241]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:30.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:29 smithi079 ceph-mon[36241]: pgmap v401: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:31.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.787+0000 7fce2b800700 1 -- 172.21.15.12:0/3249864671 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce24074cd0 msgr2=0x7fce24075150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:31.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.787+0000 7fce2b800700 1 --2- 172.21.15.12:0/3249864671 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce24074cd0 0x7fce24075150 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fce10009990 tx=0x7fce10009ca0 comp rx=0 tx=0).stop 2024-03-17T23:57:31.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.787+0000 7fce2b800700 1 -- 172.21.15.12:0/3249864671 shutdown_connections 2024-03-17T23:57:31.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.787+0000 7fce2b800700 1 --2- 172.21.15.12:0/3249864671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce2410a230 0x7fce2410c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:31.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.787+0000 7fce2b800700 1 --2- 172.21.15.12:0/3249864671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce240762a0 0x7fce24074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:31.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.787+0000 7fce2b800700 1 --2- 172.21.15.12:0/3249864671 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce24074cd0 0x7fce24075150 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:31.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.787+0000 7fce2b800700 1 -- 172.21.15.12:0/3249864671 >> 172.21.15.12:0/3249864671 conn(0x7fce240fd8f0 msgr2=0x7fce240ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:31.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.787+0000 7fce2b800700 1 -- 172.21.15.12:0/3249864671 shutdown_connections 2024-03-17T23:57:31.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.787+0000 7fce2b800700 1 -- 172.21.15.12:0/3249864671 wait complete. 2024-03-17T23:57:31.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.788+0000 7fce2b800700 1 Processor -- start 2024-03-17T23:57:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.789+0000 7fce2b800700 1 -- start start 2024-03-17T23:57:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.789+0000 7fce2b800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce24074cd0 0x7fce2419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.789+0000 7fce2b800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce240762a0 0x7fce2419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.789+0000 7fce2b800700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce2410a230 0x7fce241a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.789+0000 7fce2b800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fce24110170 con 0x7fce240762a0 2024-03-17T23:57:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.789+0000 7fce2b800700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fce2410fff0 con 0x7fce2410a230 2024-03-17T23:57:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.789+0000 7fce2b800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fce241102f0 con 0x7fce24074cd0 2024-03-17T23:57:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.789+0000 7fce29800700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce2410a230 0x7fce241a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.789+0000 7fce23e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce240762a0 0x7fce2419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.789+0000 7fce28e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce24074cd0 0x7fce2419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.789+0000 7fce29800700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce2410a230 0x7fce241a1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:58424/0 (socket says 172.21.15.12:58424) 2024-03-17T23:57:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.789+0000 7fce28e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce24074cd0 0x7fce2419a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:55006/0 (socket says 172.21.15.12:55006) 2024-03-17T23:57:31.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.789+0000 7fce29800700 1 -- 172.21.15.12:0/1094254843 learned_addr learned my addr 172.21.15.12:0/1094254843 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.790+0000 7fce29800700 1 -- 172.21.15.12:0/1094254843 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce24074cd0 msgr2=0x7fce2419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.790+0000 7fce29800700 1 --2- 172.21.15.12:0/1094254843 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce24074cd0 0x7fce2419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.790+0000 7fce29800700 1 -- 172.21.15.12:0/1094254843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce240762a0 msgr2=0x7fce2419ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.790+0000 7fce29800700 1 --2- 172.21.15.12:0/1094254843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce240762a0 0x7fce2419ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.790+0000 7fce29800700 1 -- 172.21.15.12:0/1094254843 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce241a2350 con 0x7fce2410a230 2024-03-17T23:57:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.790+0000 7fce29800700 1 --2- 172.21.15.12:0/1094254843 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce2410a230 0x7fce241a1cb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00ef30 tx=0x7fce1c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.790+0000 7fce21600700 1 -- 172.21.15.12:0/1094254843 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1c0092e0 con 0x7fce2410a230 2024-03-17T23:57:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.790+0000 7fce21600700 1 -- 172.21.15.12:0/1094254843 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce1c00ce20 con 0x7fce2410a230 2024-03-17T23:57:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.790+0000 7fce2b800700 1 -- 172.21.15.12:0/1094254843 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce241a25e0 con 0x7fce2410a230 2024-03-17T23:57:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.790+0000 7fce21600700 1 -- 172.21.15.12:0/1094254843 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1c007be0 con 0x7fce2410a230 2024-03-17T23:57:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.791+0000 7fce2b800700 1 -- 172.21.15.12:0/1094254843 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce241a2b20 con 0x7fce2410a230 2024-03-17T23:57:31.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.792+0000 7fce2b800700 1 -- 172.21.15.12:0/1094254843 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce24068f90 con 0x7fce2410a230 2024-03-17T23:57:31.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.795+0000 7fce21600700 1 -- 172.21.15.12:0/1094254843 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fce1c01b020 con 0x7fce2410a230 2024-03-17T23:57:31.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.797+0000 7fce21600700 1 --2- 172.21.15.12:0/1094254843 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fce0c0a2910 0x7fce0c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:31.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.797+0000 7fce21600700 1 -- 172.21.15.12:0/1094254843 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fce1c0ce9b0 con 0x7fce2410a230 2024-03-17T23:57:31.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.797+0000 7fce28e00700 1 --2- 172.21.15.12:0/1094254843 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fce0c0a2910 0x7fce0c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:31.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.798+0000 7fce28e00700 1 --2- 172.21.15.12:0/1094254843 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fce0c0a2910 0x7fce0c0a4dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fce18009c60 tx=0x7fce18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:31.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:31.801+0000 7fce21600700 1 -- 172.21.15.12:0/1094254843 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fce1c010040 con 0x7fce2410a230 2024-03-17T23:57:31.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:31 smithi012 ceph-mon[31541]: pgmap v402: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:32.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.114+0000 7fce2b800700 1 -- 172.21.15.12:0/1094254843 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fce24062ee0 con 0x7fce2410a230 2024-03-17T23:57:32.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.114+0000 7fce21600700 1 -- 172.21.15.12:0/1094254843 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3094 (secure 0 0 0) 0x7fce1c004ad0 con 0x7fce2410a230 2024-03-17T23:57:32.116 INFO:teuthology.orchestra.run.smithi012.stdout:e28 2024-03-17T23:57:32.116 INFO:teuthology.orchestra.run.smithi012.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-17T23:57:32.117 INFO:teuthology.orchestra.run.smithi012.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-17T23:57:32.117 INFO:teuthology.orchestra.run.smithi012.stdout:legacy client fscid: 1 2024-03-17T23:57:32.117 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:32.117 INFO:teuthology.orchestra.run.smithi012.stdout:Filesystem 'cephfs' (1) 2024-03-17T23:57:32.117 INFO:teuthology.orchestra.run.smithi012.stdout:fs_name cephfs 2024-03-17T23:57:32.117 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 28 2024-03-17T23:57:32.117 INFO:teuthology.orchestra.run.smithi012.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-17T23:57:32.117 INFO:teuthology.orchestra.run.smithi012.stdout:created 2024-03-17T23:54:56.312554+0000 2024-03-17T23:57:32.117 INFO:teuthology.orchestra.run.smithi012.stdout:modified 2024-03-17T23:55:28.030481+0000 2024-03-17T23:57:32.117 INFO:teuthology.orchestra.run.smithi012.stdout:tableserver 0 2024-03-17T23:57:32.117 INFO:teuthology.orchestra.run.smithi012.stdout:root 0 2024-03-17T23:57:32.118 INFO:teuthology.orchestra.run.smithi012.stdout:session_timeout 300 2024-03-17T23:57:32.118 INFO:teuthology.orchestra.run.smithi012.stdout:session_autoclose 300 2024-03-17T23:57:32.118 INFO:teuthology.orchestra.run.smithi012.stdout:max_file_size 1099511627776 2024-03-17T23:57:32.118 INFO:teuthology.orchestra.run.smithi012.stdout:max_xattr_size 65536 2024-03-17T23:57:32.118 INFO:teuthology.orchestra.run.smithi012.stdout:required_client_features {} 2024-03-17T23:57:32.118 INFO:teuthology.orchestra.run.smithi012.stdout:last_failure 0 2024-03-17T23:57:32.118 INFO:teuthology.orchestra.run.smithi012.stdout:last_failure_osd_epoch 0 2024-03-17T23:57:32.118 INFO:teuthology.orchestra.run.smithi012.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-17T23:57:32.118 INFO:teuthology.orchestra.run.smithi012.stdout:max_mds 5 2024-03-17T23:57:32.118 INFO:teuthology.orchestra.run.smithi012.stdout:in 0,1,2,3,4 2024-03-17T23:57:32.118 INFO:teuthology.orchestra.run.smithi012.stdout:up {0=24587,1=24575,2=24530,3=24529,4=24514} 2024-03-17T23:57:32.118 INFO:teuthology.orchestra.run.smithi012.stdout:failed 2024-03-17T23:57:32.118 INFO:teuthology.orchestra.run.smithi012.stdout:damaged 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:stopped 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:data_pools [3,4] 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:metadata_pool 2 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:inline_data disabled 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:balancer 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:bal_rank_mask -1 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:standby_count_wanted 0 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:qdb_cluster leader: 24587 members: 24575,24587,24530,24529,24514 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.b{0:24587} state up:active seq 3 addr [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] compat {c=[1],r=[1],i=[1fff]}] 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.e{0:24503} state up:standby-replay seq 1 addr [v2:172.21.15.79:6834/4099928196,v1:172.21.15.79:6835/4099928196] compat {c=[1],r=[1],i=[1fff]}] 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.a{1:24575} state up:active seq 11 addr [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] compat {c=[1],r=[1],i=[1fff]}] 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.f{1:24502} state up:standby-replay seq 1 addr [v2:172.21.15.191:6834/3192827956,v1:172.21.15.191:6835/3192827956] compat {c=[1],r=[1],i=[1fff]}] 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.h{2:24530} state up:active seq 14 addr [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] compat {c=[1],r=[1],i=[1fff]}] 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.d{2:24481} state up:standby-replay seq 1 addr [v2:172.21.15.12:6836/1104379442,v1:172.21.15.12:6837/1104379442] compat {c=[1],r=[1],i=[1fff]}] 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.i{3:24529} state up:active seq 13 addr [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] compat {c=[1],r=[1],i=[1fff]}] 2024-03-17T23:57:32.119 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.k{3:24455} state up:standby-replay seq 1 addr [v2:172.21.15.79:6832/843208408,v1:172.21.15.79:6833/843208408] compat {c=[1],r=[1],i=[1fff]}] 2024-03-17T23:57:32.120 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.c{4:24514} state up:active seq 15 addr [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] compat {c=[1],r=[1],i=[1fff]}] 2024-03-17T23:57:32.120 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.l{4:24439} state up:standby-replay seq 1 addr [v2:172.21.15.191:6832/1787103083,v1:172.21.15.191:6833/1787103083] compat {c=[1],r=[1],i=[1fff]}] 2024-03-17T23:57:32.120 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:32.120 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:32.120 INFO:teuthology.orchestra.run.smithi012.stdout:Standby daemons: 2024-03-17T23:57:32.120 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:32.120 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.j{-1:14376} state up:standby seq 1 addr [v2:172.21.15.12:6834/1091924746,v1:172.21.15.12:6835/1091924746] compat {c=[1],r=[1],i=[1fff]}] 2024-03-17T23:57:32.120 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.g{-1:14442} state up:standby seq 1 addr [v2:172.21.15.12:6838/1516405819,v1:172.21.15.12:6839/1516405819] compat {c=[1],r=[1],i=[1fff]}] 2024-03-17T23:57:32.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.117+0000 7fce2b800700 1 -- 172.21.15.12:0/1094254843 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fce0c0a2910 msgr2=0x7fce0c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:32.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.117+0000 7fce2b800700 1 --2- 172.21.15.12:0/1094254843 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fce0c0a2910 0x7fce0c0a4dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fce18009c60 tx=0x7fce18009260 comp rx=0 tx=0).stop 2024-03-17T23:57:32.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.117+0000 7fce2b800700 1 -- 172.21.15.12:0/1094254843 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce2410a230 msgr2=0x7fce241a1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:32.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.117+0000 7fce2b800700 1 --2- 172.21.15.12:0/1094254843 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce2410a230 0x7fce241a1cb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00ef30 tx=0x7fce1c00c3a0 comp rx=0 tx=0).stop 2024-03-17T23:57:32.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.117+0000 7fce2b800700 1 -- 172.21.15.12:0/1094254843 shutdown_connections 2024-03-17T23:57:32.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.117+0000 7fce2b800700 1 --2- 172.21.15.12:0/1094254843 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fce0c0a2910 0x7fce0c0a4dd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:32.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.117+0000 7fce2b800700 1 --2- 172.21.15.12:0/1094254843 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce2410a230 0x7fce241a1cb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:32.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.117+0000 7fce2b800700 1 --2- 172.21.15.12:0/1094254843 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce24074cd0 0x7fce2419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:32.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.117+0000 7fce2b800700 1 --2- 172.21.15.12:0/1094254843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce240762a0 0x7fce2419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:32.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.117+0000 7fce2b800700 1 -- 172.21.15.12:0/1094254843 >> 172.21.15.12:0/1094254843 conn(0x7fce240fd8f0 msgr2=0x7fce24106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:32.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.118+0000 7fce2b800700 1 -- 172.21.15.12:0/1094254843 shutdown_connections 2024-03-17T23:57:32.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:32.118+0000 7fce2b800700 1 -- 172.21.15.12:0/1094254843 wait complete. 2024-03-17T23:57:32.122 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 28 2024-03-17T23:57:32.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:31 smithi079 ceph-mon[36241]: pgmap v402: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:32.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:31 smithi191 ceph-mon[36708]: pgmap v402: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:33.110 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-17T23:57:33.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:32 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1094254843' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-17T23:57:33.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:32 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1094254843' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-17T23:57:33.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:32 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1094254843' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-17T23:57:33.611 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:57:34.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:33 smithi012 ceph-mon[31541]: pgmap v403: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:34.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:33 smithi079 ceph-mon[36241]: pgmap v403: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:34.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:33 smithi191 ceph-mon[36708]: pgmap v403: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:35.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:35 smithi012 ceph-mon[31541]: pgmap v404: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.960+0000 7fef20600700 1 -- 172.21.15.12:0/1056513347 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef18103510 msgr2=0x7fef18103970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.960+0000 7fef20600700 1 --2- 172.21.15.12:0/1056513347 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef18103510 0x7fef18103970 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fef040099c0 tx=0x7fef04009cd0 comp rx=0 tx=0).stop 2024-03-17T23:57:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.961+0000 7fef20600700 1 -- 172.21.15.12:0/1056513347 shutdown_connections 2024-03-17T23:57:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.961+0000 7fef20600700 1 --2- 172.21.15.12:0/1056513347 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef18103eb0 0x7fef1810a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.961+0000 7fef20600700 1 --2- 172.21.15.12:0/1056513347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef18102310 0x7fef18102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.961+0000 7fef20600700 1 --2- 172.21.15.12:0/1056513347 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef18103510 0x7fef18103970 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.961+0000 7fef20600700 1 -- 172.21.15.12:0/1056513347 >> 172.21.15.12:0/1056513347 conn(0x7fef180fd870 msgr2=0x7fef180ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.961+0000 7fef20600700 1 -- 172.21.15.12:0/1056513347 shutdown_connections 2024-03-17T23:57:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.961+0000 7fef20600700 1 -- 172.21.15.12:0/1056513347 wait complete. 2024-03-17T23:57:35.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.962+0000 7fef20600700 1 Processor -- start 2024-03-17T23:57:35.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.962+0000 7fef20600700 1 -- start start 2024-03-17T23:57:35.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.963+0000 7fef20600700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef18102310 0x7fef18076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:35.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.963+0000 7fef20600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef18103510 0x7fef180746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:35.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.963+0000 7fef20600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef18103eb0 0x7fef18074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:35.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.963+0000 7fef20600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fef1810df30 con 0x7fef18103510 2024-03-17T23:57:35.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.963+0000 7fef20600700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fef1810ddb0 con 0x7fef18102310 2024-03-17T23:57:35.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.963+0000 7fef20600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fef1810e0b0 con 0x7fef18103eb0 2024-03-17T23:57:35.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.963+0000 7fef1dc00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef18102310 0x7fef18076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:35.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.963+0000 7fef1dc00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef18102310 0x7fef18076070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36238/0 (socket says 172.21.15.12:36238) 2024-03-17T23:57:35.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.963+0000 7fef1e600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef18103eb0 0x7fef18074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:35.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.963+0000 7fef1dc00700 1 -- 172.21.15.12:0/397845680 learned_addr learned my addr 172.21.15.12:0/397845680 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:35.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.964+0000 7fef1dc00700 1 -- 172.21.15.12:0/397845680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef18103eb0 msgr2=0x7fef18074c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:35.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.964+0000 7fef1dc00700 1 --2- 172.21.15.12:0/397845680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef18103eb0 0x7fef18074c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:35.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.964+0000 7fef1dc00700 1 -- 172.21.15.12:0/397845680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef18103510 msgr2=0x7fef180746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.964+0000 7fef1dc00700 1 --2- 172.21.15.12:0/397845680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef18103510 0x7fef180746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.964+0000 7fef1dc00700 1 -- 172.21.15.12:0/397845680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef180754b0 con 0x7fef18102310 2024-03-17T23:57:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.964+0000 7fef1dc00700 1 --2- 172.21.15.12:0/397845680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef18102310 0x7fef18076070 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00e970 tx=0x7fef0c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.964+0000 7fef16000700 1 -- 172.21.15.12:0/397845680 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c00cbd0 con 0x7fef18102310 2024-03-17T23:57:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.964+0000 7fef16000700 1 -- 172.21.15.12:0/397845680 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef0c004500 con 0x7fef18102310 2024-03-17T23:57:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.964+0000 7fef20600700 1 -- 172.21.15.12:0/397845680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef181a2700 con 0x7fef18102310 2024-03-17T23:57:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.964+0000 7fef16000700 1 -- 172.21.15.12:0/397845680 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c003e30 con 0x7fef18102310 2024-03-17T23:57:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.965+0000 7fef20600700 1 -- 172.21.15.12:0/397845680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef181a2c40 con 0x7fef18102310 2024-03-17T23:57:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.966+0000 7fef20600700 1 -- 172.21.15.12:0/397845680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef18068f90 con 0x7fef18102310 2024-03-17T23:57:35.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.967+0000 7fef16000700 1 -- 172.21.15.12:0/397845680 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fef0c004020 con 0x7fef18102310 2024-03-17T23:57:35.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.969+0000 7fef16000700 1 --2- 172.21.15.12:0/397845680 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fef000a2910 0x7fef000a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.969+0000 7fef16000700 1 -- 172.21.15.12:0/397845680 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fef0c0cf0e0 con 0x7fef18102310 2024-03-17T23:57:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.969+0000 7fef1d200700 1 --2- 172.21.15.12:0/397845680 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fef000a2910 0x7fef000a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.970+0000 7fef1d200700 1 --2- 172.21.15.12:0/397845680 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fef000a2910 0x7fef000a4dd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fef040098e0 tx=0x7fef04000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:35.972+0000 7fef16000700 1 -- 172.21.15.12:0/397845680 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fef0c00b2c0 con 0x7fef18102310 2024-03-17T23:57:36.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.201+0000 7fef20600700 1 -- 172.21.15.12:0/397845680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fef1802db70 con 0x7fef18102310 2024-03-17T23:57:36.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.202+0000 7fef16000700 1 -- 172.21.15.12:0/397845680 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v94) ==== 56+0+5600 (secure 0 0 0) 0x7fef0c09bda0 con 0x7fef18102310 2024-03-17T23:57:36.204 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 94 2024-03-17T23:57:36.204 INFO:teuthology.orchestra.run.smithi012.stdout:fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-17T23:57:36.205 INFO:teuthology.orchestra.run.smithi012.stdout:created 2024-03-17T23:45:45.898190+0000 2024-03-17T23:57:36.205 INFO:teuthology.orchestra.run.smithi012.stdout:modified 2024-03-17T23:56:55.719277+0000 2024-03-17T23:57:36.205 INFO:teuthology.orchestra.run.smithi012.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-17T23:57:36.205 INFO:teuthology.orchestra.run.smithi012.stdout:crush_version 27 2024-03-17T23:57:36.205 INFO:teuthology.orchestra.run.smithi012.stdout:full_ratio 0.95 2024-03-17T23:57:36.205 INFO:teuthology.orchestra.run.smithi012.stdout:backfillfull_ratio 0.9 2024-03-17T23:57:36.205 INFO:teuthology.orchestra.run.smithi012.stdout:nearfull_ratio 0.85 2024-03-17T23:57:36.205 INFO:teuthology.orchestra.run.smithi012.stdout:require_min_compat_client luminous 2024-03-17T23:57:36.205 INFO:teuthology.orchestra.run.smithi012.stdout:min_compat_client luminous 2024-03-17T23:57:36.205 INFO:teuthology.orchestra.run.smithi012.stdout:require_osd_release squid 2024-03-17T23:57:36.205 INFO:teuthology.orchestra.run.smithi012.stdout:stretch_mode_enabled false 2024-03-17T23:57:36.206 INFO:teuthology.orchestra.run.smithi012.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-17T23:57:36.206 INFO:teuthology.orchestra.run.smithi012.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-17T23:57:36.206 INFO:teuthology.orchestra.run.smithi012.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-17T23:57:36.206 INFO:teuthology.orchestra.run.smithi012.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-17T23:57:36.206 INFO:teuthology.orchestra.run.smithi012.stdout:max_osd 12 2024-03-17T23:57:36.206 INFO:teuthology.orchestra.run.smithi012.stdout:osd.0 up in weight 1 up_from 8 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6802/3127075279,v1:172.21.15.12:6803/3127075279] [v2:172.21.15.12:6804/3127075279,v1:172.21.15.12:6805/3127075279] exists,up b91e248b-f111-4dc6-9dc8-07fd0195699a 2024-03-17T23:57:36.206 INFO:teuthology.orchestra.run.smithi012.stdout:osd.1 up in weight 1 up_from 13 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6800/890307857,v1:172.21.15.79:6801/890307857] [v2:172.21.15.79:6802/890307857,v1:172.21.15.79:6803/890307857] exists,up 13bf3ff0-c8b3-4c24-9960-c6568256772f 2024-03-17T23:57:36.206 INFO:teuthology.orchestra.run.smithi012.stdout:osd.2 up in weight 1 up_from 18 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6800/936135993,v1:172.21.15.191:6801/936135993] [v2:172.21.15.191:6802/936135993,v1:172.21.15.191:6803/936135993] exists,up 24aaf834-2a6b-4bb0-986b-3fe9525ecacd 2024-03-17T23:57:36.206 INFO:teuthology.orchestra.run.smithi012.stdout:osd.3 up in weight 1 up_from 25 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6810/769150575,v1:172.21.15.12:6811/769150575] [v2:172.21.15.12:6812/769150575,v1:172.21.15.12:6813/769150575] exists,up ba4c0892-f619-45d4-a91a-00a2b0cc3d78 2024-03-17T23:57:36.206 INFO:teuthology.orchestra.run.smithi012.stdout:osd.4 up in weight 1 up_from 33 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6808/3381761360,v1:172.21.15.79:6809/3381761360] [v2:172.21.15.79:6810/3381761360,v1:172.21.15.79:6811/3381761360] exists,up ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a 2024-03-17T23:57:36.206 INFO:teuthology.orchestra.run.smithi012.stdout:osd.5 up in weight 1 up_from 38 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6808/862706194,v1:172.21.15.191:6809/862706194] [v2:172.21.15.191:6810/862706194,v1:172.21.15.191:6811/862706194] exists,up e3a56fd4-99df-4230-a51f-382af53afef2 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:osd.6 up in weight 1 up_from 46 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6818/3661372073,v1:172.21.15.12:6819/3661372073] [v2:172.21.15.12:6820/3661372073,v1:172.21.15.12:6821/3661372073] exists,up 309e6139-7f6e-4a92-bce4-e240cdc72f94 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:osd.7 up in weight 1 up_from 51 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6816/1956068856,v1:172.21.15.79:6817/1956068856] [v2:172.21.15.79:6818/1956068856,v1:172.21.15.79:6819/1956068856] exists,up 9f77cf43-cd10-4ef1-bb88-dbe4f154c353 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:osd.8 up in weight 1 up_from 59 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6816/704430703,v1:172.21.15.191:6817/704430703] [v2:172.21.15.191:6818/704430703,v1:172.21.15.191:6819/704430703] exists,up 9ae081be-d65c-456a-81f5-8a299254b8ae 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:osd.9 up in weight 1 up_from 67 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6826/140672796,v1:172.21.15.12:6827/140672796] [v2:172.21.15.12:6828/140672796,v1:172.21.15.12:6829/140672796] exists,up f8a4b561-6262-4a05-855f-086c7d39a955 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:osd.10 up in weight 1 up_from 72 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6824/3620799359,v1:172.21.15.79:6825/3620799359] [v2:172.21.15.79:6826/3620799359,v1:172.21.15.79:6827/3620799359] exists,up 4d562fe1-e7a4-41f6-81e9-95f8745539bd 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:osd.11 up in weight 1 up_from 80 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6824/1030815303,v1:172.21.15.191:6825/1030815303] [v2:172.21.15.191:6826/1030815303,v1:172.21.15.191:6827/1030815303] exists,up 2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 2.5 [8,11] 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 2.10 [2,4] 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 2.13 [2,11] 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 2.1a [8,1] 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.1 [4,9] 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.3 [4,9] 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.e [5,9] 2024-03-17T23:57:36.207 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 4.6 [9,11] 2024-03-17T23:57:36.208 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 4.12 [5,10] 2024-03-17T23:57:36.208 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 4.13 [9,8] 2024-03-17T23:57:36.208 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 4.1f [9,8] 2024-03-17T23:57:36.208 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6801/2526360059 expires 2024-03-18T23:46:53.955890+0000 2024-03-17T23:57:36.208 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/2336882461 expires 2024-03-18T23:46:53.955890+0000 2024-03-17T23:57:36.208 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6800/1563339918 expires 2024-03-18T23:46:19.706199+0000 2024-03-17T23:57:36.208 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/1976882257 expires 2024-03-18T23:46:53.955890+0000 2024-03-17T23:57:36.208 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/501542164 expires 2024-03-18T23:46:53.955890+0000 2024-03-17T23:57:36.208 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/654354506 expires 2024-03-18T23:46:19.706199+0000 2024-03-17T23:57:36.208 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6801/1563339918 expires 2024-03-18T23:46:19.706199+0000 2024-03-17T23:57:36.208 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/1974593432 expires 2024-03-18T23:46:19.706199+0000 2024-03-17T23:57:36.208 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6800/2526360059 expires 2024-03-18T23:46:53.955890+0000 2024-03-17T23:57:36.208 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/1690054132 expires 2024-03-18T23:46:19.706199+0000 2024-03-17T23:57:36.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.205+0000 7fef20600700 1 -- 172.21.15.12:0/397845680 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fef000a2910 msgr2=0x7fef000a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:36.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.205+0000 7fef20600700 1 --2- 172.21.15.12:0/397845680 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fef000a2910 0x7fef000a4dd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fef040098e0 tx=0x7fef04000bc0 comp rx=0 tx=0).stop 2024-03-17T23:57:36.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.205+0000 7fef20600700 1 -- 172.21.15.12:0/397845680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef18102310 msgr2=0x7fef18076070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:36.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.205+0000 7fef20600700 1 --2- 172.21.15.12:0/397845680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef18102310 0x7fef18076070 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00e970 tx=0x7fef0c00ec80 comp rx=0 tx=0).stop 2024-03-17T23:57:36.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.205+0000 7fef20600700 1 -- 172.21.15.12:0/397845680 shutdown_connections 2024-03-17T23:57:36.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.205+0000 7fef20600700 1 --2- 172.21.15.12:0/397845680 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fef000a2910 0x7fef000a4dd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:36.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.205+0000 7fef20600700 1 --2- 172.21.15.12:0/397845680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef18103eb0 0x7fef18074c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:36.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.205+0000 7fef20600700 1 --2- 172.21.15.12:0/397845680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef18102310 0x7fef18076070 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:36.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.205+0000 7fef20600700 1 --2- 172.21.15.12:0/397845680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef18103510 0x7fef180746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:36.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.205+0000 7fef20600700 1 -- 172.21.15.12:0/397845680 >> 172.21.15.12:0/397845680 conn(0x7fef180fd870 msgr2=0x7fef18106810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:36.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.205+0000 7fef20600700 1 -- 172.21.15.12:0/397845680 shutdown_connections 2024-03-17T23:57:36.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:36.206+0000 7fef20600700 1 -- 172.21.15.12:0/397845680 wait complete. 2024-03-17T23:57:36.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:35 smithi079 ceph-mon[36241]: pgmap v404: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:36.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:35 smithi191 ceph-mon[36708]: pgmap v404: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:36.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:36 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/397845680' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-17T23:57:37.234 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-17T23:57:37.247 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:57:37.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:36 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/397845680' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-17T23:57:37.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:36 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/397845680' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-17T23:57:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.441+0000 7fd09d600640 1 -- 172.21.15.12:0/2826254347 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0981017e0 msgr2=0x7fd098101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.441+0000 7fd09d600640 1 --2- 172.21.15.12:0/2826254347 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0981017e0 0x7fd098101c00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd088009860 tx=0x7fd08802bb70 comp rx=0 tx=0).stop 2024-03-17T23:57:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.441+0000 7fd09d600640 1 -- 172.21.15.12:0/2826254347 shutdown_connections 2024-03-17T23:57:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.441+0000 7fd09d600640 1 --2- 172.21.15.12:0/2826254347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd098103390 0x7fd098109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.441+0000 7fd09d600640 1 --2- 172.21.15.12:0/2826254347 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0981029d0 0x7fd098102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.441+0000 7fd09d600640 1 --2- 172.21.15.12:0/2826254347 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0981017e0 0x7fd098101c00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.441+0000 7fd09d600640 1 -- 172.21.15.12:0/2826254347 >> 172.21.15.12:0/2826254347 conn(0x7fd0980fcd50 msgr2=0x7fd0980ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.441+0000 7fd09d600640 1 -- 172.21.15.12:0/2826254347 shutdown_connections 2024-03-17T23:57:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.441+0000 7fd09d600640 1 -- 172.21.15.12:0/2826254347 wait complete. 2024-03-17T23:57:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.442+0000 7fd09d600640 1 Processor -- start 2024-03-17T23:57:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.442+0000 7fd09d600640 1 -- start start 2024-03-17T23:57:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd09d600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0981017e0 0x7fd098075540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd09d600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0981029d0 0x7fd098073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd09d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd098103390 0x7fd098074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd09d600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd09810d450 con 0x7fd0981029d0 2024-03-17T23:57:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd09d600640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd09810d2d0 con 0x7fd0981017e0 2024-03-17T23:57:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd09d600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd09810d5d0 con 0x7fd098103390 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd096a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0981017e0 0x7fd098075540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd096a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0981017e0 0x7fd098075540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36258/0 (socket says 172.21.15.12:36258) 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd096a00640 1 -- 172.21.15.12:0/1468483111 learned_addr learned my addr 172.21.15.12:0/1468483111 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd096000640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0981029d0 0x7fd098073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd097400640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd098103390 0x7fd098074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd096a00640 1 -- 172.21.15.12:0/1468483111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd098103390 msgr2=0x7fd098074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd096a00640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd098103390 0x7fd098074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd096a00640 1 -- 172.21.15.12:0/1468483111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0981029d0 msgr2=0x7fd098073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd096a00640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0981029d0 0x7fd098073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.443+0000 7fd096a00640 1 -- 172.21.15.12:0/1468483111 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0980749d0 con 0x7fd0981017e0 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.444+0000 7fd096a00640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0981017e0 0x7fd098075540 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd0880349e0 tx=0x7fd088032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.444+0000 7fd08f400640 1 -- 172.21.15.12:0/1468483111 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088041020 con 0x7fd0981017e0 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.444+0000 7fd08f400640 1 -- 172.21.15.12:0/1468483111 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd08802c9f0 con 0x7fd0981017e0 2024-03-17T23:57:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.444+0000 7fd08f400640 1 -- 172.21.15.12:0/1468483111 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088039040 con 0x7fd0981017e0 2024-03-17T23:57:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.444+0000 7fd09d600640 1 -- 172.21.15.12:0/1468483111 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0981a1c70 con 0x7fd0981017e0 2024-03-17T23:57:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.444+0000 7fd09d600640 1 -- 172.21.15.12:0/1468483111 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0981a2130 con 0x7fd0981017e0 2024-03-17T23:57:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.446+0000 7fd09d600640 1 -- 172.21.15.12:0/1468483111 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd098108c20 con 0x7fd0981017e0 2024-03-17T23:57:37.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.447+0000 7fd08f400640 1 -- 172.21.15.12:0/1468483111 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd088044020 con 0x7fd0981017e0 2024-03-17T23:57:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.448+0000 7fd08f400640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd07c0a2800 0x7fd07c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.449+0000 7fd08f400640 1 -- 172.21.15.12:0/1468483111 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fd0880f0650 con 0x7fd0981017e0 2024-03-17T23:57:37.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.449+0000 7fd096000640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd07c0a2800 0x7fd07c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:37.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.449+0000 7fd096000640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd07c0a2800 0x7fd07c0a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd0980752b0 tx=0x7fd084005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:37.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.452+0000 7fd08f400640 1 -- 172.21.15.12:0/1468483111 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd08804b020 con 0x7fd0981017e0 2024-03-17T23:57:37.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.712+0000 7fd09d600640 1 -- 172.21.15.12:0/1468483111 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0980623e0 con 0x7fd0981017e0 2024-03-17T23:57:37.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.714+0000 7fd08f400640 1 -- 172.21.15.12:0/1468483111 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24618 (secure 0 0 0) 0x7fd0880bd310 con 0x7fd0981017e0 2024-03-17T23:57:37.716 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:37.716 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":94,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:56:55.719277+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:57:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.717+0000 7fd09d600640 1 -- 172.21.15.12:0/1468483111 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd07c0a2800 msgr2=0x7fd07c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.717+0000 7fd09d600640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd07c0a2800 0x7fd07c0a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd0980752b0 tx=0x7fd084005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.717+0000 7fd09d600640 1 -- 172.21.15.12:0/1468483111 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0981017e0 msgr2=0x7fd098075540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.717+0000 7fd09d600640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0981017e0 0x7fd098075540 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd0880349e0 tx=0x7fd088032430 comp rx=0 tx=0).stop 2024-03-17T23:57:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.717+0000 7fd09d600640 1 -- 172.21.15.12:0/1468483111 shutdown_connections 2024-03-17T23:57:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.717+0000 7fd09d600640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd07c0a2800 0x7fd07c0a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.717+0000 7fd09d600640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd098103390 0x7fd098074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.717+0000 7fd09d600640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0981029d0 0x7fd098073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.717+0000 7fd09d600640 1 --2- 172.21.15.12:0/1468483111 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0981017e0 0x7fd098075540 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.717+0000 7fd09d600640 1 -- 172.21.15.12:0/1468483111 >> 172.21.15.12:0/1468483111 conn(0x7fd0980fcd50 msgr2=0x7fd098105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.717+0000 7fd09d600640 1 -- 172.21.15.12:0/1468483111 shutdown_connections 2024-03-17T23:57:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.717+0000 7fd09d600640 1 -- 172.21.15.12:0/1468483111 wait complete. 2024-03-17T23:57:37.737 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:49:38.640282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:54:54.540345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:54:55.803926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:55:00.355066+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:57:37.737 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:57:37.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.931+0000 7f5297e00640 1 -- 172.21.15.12:0/663778084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52981016e0 msgr2=0x7f5298101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.931+0000 7f5297e00640 1 --2- 172.21.15.12:0/663778084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52981016e0 0x7f5298101b00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5288009900 tx=0x7f528802bc40 comp rx=0 tx=0).stop 2024-03-17T23:57:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.931+0000 7f5297e00640 1 -- 172.21.15.12:0/663778084 shutdown_connections 2024-03-17T23:57:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.931+0000 7f5297e00640 1 --2- 172.21.15.12:0/663778084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5298103290 0x7f5298109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.931+0000 7f5297e00640 1 --2- 172.21.15.12:0/663778084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52981028d0 0x7f5298102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.931+0000 7f5297e00640 1 --2- 172.21.15.12:0/663778084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52981016e0 0x7f5298101b00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.931+0000 7f5297e00640 1 -- 172.21.15.12:0/663778084 >> 172.21.15.12:0/663778084 conn(0x7f52980fcc50 msgr2=0x7f52980ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.931+0000 7f5297e00640 1 -- 172.21.15.12:0/663778084 shutdown_connections 2024-03-17T23:57:37.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.931+0000 7f5297e00640 1 -- 172.21.15.12:0/663778084 wait complete. 2024-03-17T23:57:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.932+0000 7f5297e00640 1 Processor -- start 2024-03-17T23:57:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.932+0000 7f5297e00640 1 -- start start 2024-03-17T23:57:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5297e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52981016e0 0x7f5298199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5297e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52981028d0 0x7f529819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5297e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5298103290 0x7f52981a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5297e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f529810d380 con 0x7f5298103290 2024-03-17T23:57:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5297e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f529810d200 con 0x7f52981028d0 2024-03-17T23:57:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5297e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f529810d500 con 0x7f52981016e0 2024-03-17T23:57:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5296a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52981016e0 0x7f5298199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5296a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52981016e0 0x7f5298199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:39566/0 (socket says 172.21.15.12:39566) 2024-03-17T23:57:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5296a00640 1 -- 172.21.15.12:0/1487652116 learned_addr learned my addr 172.21.15.12:0/1487652116 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5296000640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52981028d0 0x7f529819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5297400640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5298103290 0x7f52981a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5296a00640 1 -- 172.21.15.12:0/1487652116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52981028d0 msgr2=0x7f529819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5296a00640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52981028d0 0x7f529819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5296a00640 1 -- 172.21.15.12:0/1487652116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5298103290 msgr2=0x7f52981a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5296a00640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5298103290 0x7f52981a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5296a00640 1 -- 172.21.15.12:0/1487652116 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52981a1830 con 0x7f52981016e0 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5297400640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5298103290 0x7f52981a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.933+0000 7f5296000640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52981028d0 0x7f529819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.934+0000 7f5296a00640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52981016e0 0x7f5298199c20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f528802efd0 tx=0x7f5288005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.934+0000 7f528f400640 1 -- 172.21.15.12:0/1487652116 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288041070 con 0x7f52981016e0 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.934+0000 7f528f400640 1 -- 172.21.15.12:0/1487652116 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52880044d0 con 0x7f52981016e0 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.934+0000 7f5297e00640 1 -- 172.21.15.12:0/1487652116 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52981a1ac0 con 0x7f52981016e0 2024-03-17T23:57:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.934+0000 7f528f400640 1 -- 172.21.15.12:0/1487652116 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288034440 con 0x7f52981016e0 2024-03-17T23:57:37.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.934+0000 7f5297e00640 1 -- 172.21.15.12:0/1487652116 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52981a1f80 con 0x7f52981016e0 2024-03-17T23:57:37.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.935+0000 7f5297e00640 1 -- 172.21.15.12:0/1487652116 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52981a2150 con 0x7f52981016e0 2024-03-17T23:57:37.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.937+0000 7f528f400640 1 -- 172.21.15.12:0/1487652116 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5288044070 con 0x7f52981016e0 2024-03-17T23:57:37.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.938+0000 7f528f400640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f527c0a2800 0x7f527c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:37.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.938+0000 7f528f400640 1 -- 172.21.15.12:0/1487652116 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f52880f0b10 con 0x7f52981016e0 2024-03-17T23:57:37.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.938+0000 7f5296000640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f527c0a2800 0x7f527c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:37.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.939+0000 7f5296000640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f527c0a2800 0x7f527c0a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5284009fc0 tx=0x7f5284005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:37.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:37.946+0000 7f528f400640 1 -- 172.21.15.12:0/1487652116 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f52880bd7d0 con 0x7f52981016e0 2024-03-17T23:57:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.163+0000 7f5297e00640 1 -- 172.21.15.12:0/1487652116 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f52981a2320 con 0x7f52981016e0 2024-03-17T23:57:38.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.163+0000 7f528f400640 1 -- 172.21.15.12:0/1487652116 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f52880bd0d0 con 0x7f52981016e0 2024-03-17T23:57:38.165 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-17T23:57:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.166+0000 7f5297e00640 1 -- 172.21.15.12:0/1487652116 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f527c0a2800 msgr2=0x7f527c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.166+0000 7f5297e00640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f527c0a2800 0x7f527c0a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5284009fc0 tx=0x7f5284005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.166+0000 7f5297e00640 1 -- 172.21.15.12:0/1487652116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52981016e0 msgr2=0x7f5298199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.166+0000 7f5297e00640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52981016e0 0x7f5298199c20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f528802efd0 tx=0x7f5288005d50 comp rx=0 tx=0).stop 2024-03-17T23:57:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.166+0000 7f5297e00640 1 -- 172.21.15.12:0/1487652116 shutdown_connections 2024-03-17T23:57:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.166+0000 7f5297e00640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f527c0a2800 0x7f527c0a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.166+0000 7f5297e00640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5298103290 0x7f52981a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.166+0000 7f5297e00640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52981028d0 0x7f529819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.166+0000 7f5297e00640 1 --2- 172.21.15.12:0/1487652116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52981016e0 0x7f5298199c20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.166+0000 7f5297e00640 1 -- 172.21.15.12:0/1487652116 >> 172.21.15.12:0/1487652116 conn(0x7f52980fcc50 msgr2=0x7f5298073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.166+0000 7f5297e00640 1 -- 172.21.15.12:0/1487652116 shutdown_connections 2024-03-17T23:57:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.166+0000 7f5297e00640 1 -- 172.21.15.12:0/1487652116 wait complete. 2024-03-17T23:57:38.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:37 smithi012 ceph-mon[31541]: pgmap v405: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:57:38.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:37 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1468483111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:38.184 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:57:38.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:37 smithi079 ceph-mon[36241]: pgmap v405: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:57:38.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:37 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1468483111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:38.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:37 smithi191 ceph-mon[36708]: pgmap v405: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:57:38.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:37 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1468483111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:38.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.383+0000 7f326b200640 1 -- 172.21.15.12:0/676244990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f326c1016e0 msgr2=0x7f326c101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:38.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.383+0000 7f326b200640 1 --2- 172.21.15.12:0/676244990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f326c1016e0 0x7f326c101b00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f325c009900 tx=0x7f325c02bc30 comp rx=0 tx=0).stop 2024-03-17T23:57:38.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.383+0000 7f326b200640 1 -- 172.21.15.12:0/676244990 shutdown_connections 2024-03-17T23:57:38.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.383+0000 7f326b200640 1 --2- 172.21.15.12:0/676244990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f326c103290 0x7f326c109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.383+0000 7f326b200640 1 --2- 172.21.15.12:0/676244990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1028d0 0x7f326c102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.383+0000 7f326b200640 1 --2- 172.21.15.12:0/676244990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f326c1016e0 0x7f326c101b00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.383+0000 7f326b200640 1 -- 172.21.15.12:0/676244990 >> 172.21.15.12:0/676244990 conn(0x7f326c0fcc50 msgr2=0x7f326c0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:38.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.383+0000 7f326b200640 1 -- 172.21.15.12:0/676244990 shutdown_connections 2024-03-17T23:57:38.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.383+0000 7f326b200640 1 -- 172.21.15.12:0/676244990 wait complete. 2024-03-17T23:57:38.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.384+0000 7f326b200640 1 Processor -- start 2024-03-17T23:57:38.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.384+0000 7f326b200640 1 -- start start 2024-03-17T23:57:38.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.384+0000 7f326b200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f326c1016e0 0x7f326c19e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:38.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.384+0000 7f326b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1028d0 0x7f326c19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:38.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.384+0000 7f326b200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f326c103290 0x7f326c1a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:38.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.384+0000 7f326b200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f326c074e90 con 0x7f326c103290 2024-03-17T23:57:38.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.384+0000 7f326b200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f326c074d10 con 0x7f326c1016e0 2024-03-17T23:57:38.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.384+0000 7f326b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f326c075010 con 0x7f326c1028d0 2024-03-17T23:57:38.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.384+0000 7f3269e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f326c1016e0 0x7f326c19e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:38.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.384+0000 7f3269e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f326c1016e0 0x7f326c19e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36294/0 (socket says 172.21.15.12:36294) 2024-03-17T23:57:38.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.384+0000 7f3269400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1028d0 0x7f326c19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:38.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.384+0000 7f3269e00640 1 -- 172.21.15.12:0/38858923 learned_addr learned my addr 172.21.15.12:0/38858923 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:38.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.385+0000 7f326a800640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f326c103290 0x7f326c1a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:38.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.385+0000 7f3269e00640 1 -- 172.21.15.12:0/38858923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1028d0 msgr2=0x7f326c19e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:38.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.385+0000 7f3269e00640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1028d0 0x7f326c19e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.385+0000 7f3269e00640 1 -- 172.21.15.12:0/38858923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f326c103290 msgr2=0x7f326c1a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:38.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.385+0000 7f3269e00640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f326c103290 0x7f326c1a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.385+0000 7f3269e00640 1 -- 172.21.15.12:0/38858923 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326c1a5c60 con 0x7f326c1016e0 2024-03-17T23:57:38.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.385+0000 7f326a800640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f326c103290 0x7f326c1a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:38.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.385+0000 7f3269e00640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f326c1016e0 0x7f326c19e090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f325c000c00 tx=0x7f325c005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:38.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.385+0000 7f3262a00640 1 -- 172.21.15.12:0/38858923 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c041070 con 0x7f326c1016e0 2024-03-17T23:57:38.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.385+0000 7f3262a00640 1 -- 172.21.15.12:0/38858923 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f325c0044d0 con 0x7f326c1016e0 2024-03-17T23:57:38.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.385+0000 7f3262a00640 1 -- 172.21.15.12:0/38858923 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c034440 con 0x7f326c1016e0 2024-03-17T23:57:38.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.386+0000 7f326b200640 1 -- 172.21.15.12:0/38858923 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326c1a5ef0 con 0x7f326c1016e0 2024-03-17T23:57:38.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.386+0000 7f326b200640 1 -- 172.21.15.12:0/38858923 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f326c1a62a0 con 0x7f326c1016e0 2024-03-17T23:57:38.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.387+0000 7f326b200640 1 -- 172.21.15.12:0/38858923 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f326c1a64d0 con 0x7f326c1016e0 2024-03-17T23:57:38.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.390+0000 7f3262a00640 1 -- 172.21.15.12:0/38858923 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f325c044070 con 0x7f326c1016e0 2024-03-17T23:57:38.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.391+0000 7f3262a00640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f32500a2800 0x7f32500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:38.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.392+0000 7f3262a00640 1 -- 172.21.15.12:0/38858923 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f325c0f09e0 con 0x7f326c1016e0 2024-03-17T23:57:38.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.392+0000 7f3269400640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f32500a2800 0x7f32500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:38.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.392+0000 7f3269400640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f32500a2800 0x7f32500a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f326c19f550 tx=0x7f3258005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:38.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.395+0000 7f3262a00640 1 -- 172.21.15.12:0/38858923 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f325c0bd6a0 con 0x7f326c1016e0 2024-03-17T23:57:38.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.595+0000 7f326b200640 1 -- 172.21.15.12:0/38858923 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f326c1a6d30 con 0x7f326c1016e0 2024-03-17T23:57:38.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.596+0000 7f3262a00640 1 -- 172.21.15.12:0/38858923 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f325c0bcfa0 con 0x7f326c1016e0 2024-03-17T23:57:38.598 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-17T23:57:38.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.599+0000 7f326b200640 1 -- 172.21.15.12:0/38858923 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f32500a2800 msgr2=0x7f32500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:38.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.599+0000 7f326b200640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f32500a2800 0x7f32500a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f326c19f550 tx=0x7f3258005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:38.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.599+0000 7f326b200640 1 -- 172.21.15.12:0/38858923 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f326c1016e0 msgr2=0x7f326c19e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:38.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.599+0000 7f326b200640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f326c1016e0 0x7f326c19e090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f325c000c00 tx=0x7f325c005d50 comp rx=0 tx=0).stop 2024-03-17T23:57:38.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.599+0000 7f326b200640 1 -- 172.21.15.12:0/38858923 shutdown_connections 2024-03-17T23:57:38.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.599+0000 7f326b200640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f32500a2800 0x7f32500a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.599+0000 7f326b200640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f326c103290 0x7f326c1a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.599+0000 7f326b200640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1028d0 0x7f326c19e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.599+0000 7f326b200640 1 --2- 172.21.15.12:0/38858923 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f326c1016e0 0x7f326c19e090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.599+0000 7f326b200640 1 -- 172.21.15.12:0/38858923 >> 172.21.15.12:0/38858923 conn(0x7f326c0fcc50 msgr2=0x7f326c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:38.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.599+0000 7f326b200640 1 -- 172.21.15.12:0/38858923 shutdown_connections 2024-03-17T23:57:38.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.600+0000 7f326b200640 1 -- 172.21.15.12:0/38858923 wait complete. 2024-03-17T23:57:38.624 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:57:38.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.858+0000 7f9bace00640 1 -- 172.21.15.12:0/2567970991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba81008a0 msgr2=0x7f9ba8100d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:38.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.858+0000 7f9bace00640 1 --2- 172.21.15.12:0/2567970991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba81008a0 0x7f9ba8100d00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009900 tx=0x7f9b9002bc40 comp rx=0 tx=0).stop 2024-03-17T23:57:38.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.858+0000 7f9bace00640 1 -- 172.21.15.12:0/2567970991 shutdown_connections 2024-03-17T23:57:38.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.858+0000 7f9bace00640 1 --2- 172.21.15.12:0/2567970991 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ba8101240 0x7f9ba81016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.858+0000 7f9bace00640 1 --2- 172.21.15.12:0/2567970991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba81008a0 0x7f9ba8100d00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.858+0000 7f9bace00640 1 --2- 172.21.15.12:0/2567970991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba80fff40 0x7f9ba8100360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.858+0000 7f9bace00640 1 -- 172.21.15.12:0/2567970991 >> 172.21.15.12:0/2567970991 conn(0x7f9ba80fbb60 msgr2=0x7f9ba80fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:38.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.858+0000 7f9bace00640 1 -- 172.21.15.12:0/2567970991 shutdown_connections 2024-03-17T23:57:38.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.858+0000 7f9bace00640 1 -- 172.21.15.12:0/2567970991 wait complete. 2024-03-17T23:57:38.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.859+0000 7f9bace00640 1 Processor -- start 2024-03-17T23:57:38.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.860+0000 7f9bace00640 1 -- start start 2024-03-17T23:57:38.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.860+0000 7f9bace00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba80fff40 0x7f9ba8199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:38.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.860+0000 7f9bace00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ba81008a0 0x7f9ba819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:38.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.860+0000 7f9bace00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba8101240 0x7f9ba81a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:38.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.860+0000 7f9bace00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba810f510 con 0x7f9ba8101240 2024-03-17T23:57:38.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.860+0000 7f9bace00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba810f390 con 0x7f9ba81008a0 2024-03-17T23:57:38.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.860+0000 7f9bace00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba810f690 con 0x7f9ba80fff40 2024-03-17T23:57:38.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.860+0000 7f9ba7400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba80fff40 0x7f9ba8199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:38.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.860+0000 7f9b9fe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ba81008a0 0x7f9ba819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:38.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.860+0000 7f9ba7e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba8101240 0x7f9ba81a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:38.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.860+0000 7f9ba7400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba80fff40 0x7f9ba8199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:39598/0 (socket says 172.21.15.12:39598) 2024-03-17T23:57:38.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.860+0000 7f9ba7400640 1 -- 172.21.15.12:0/1043678242 learned_addr learned my addr 172.21.15.12:0/1043678242 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:38.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.861+0000 7f9ba7400640 1 -- 172.21.15.12:0/1043678242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ba81008a0 msgr2=0x7f9ba819a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:38.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.861+0000 7f9ba7400640 1 --2- 172.21.15.12:0/1043678242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ba81008a0 0x7f9ba819a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.861+0000 7f9ba7400640 1 -- 172.21.15.12:0/1043678242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba8101240 msgr2=0x7f9ba81a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:38.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.861+0000 7f9ba7400640 1 --2- 172.21.15.12:0/1043678242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba8101240 0x7f9ba81a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:38.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.861+0000 7f9ba7400640 1 -- 172.21.15.12:0/1043678242 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba81a1800 con 0x7f9ba80fff40 2024-03-17T23:57:38.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.861+0000 7f9b9fe00640 1 --2- 172.21.15.12:0/1043678242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ba81008a0 0x7f9ba819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:38.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.861+0000 7f9ba7400640 1 --2- 172.21.15.12:0/1043678242 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba80fff40 0x7f9ba8199be0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800e9f0 tx=0x7f9b9800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:38.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.861+0000 7f9ba4c00640 1 -- 172.21.15.12:0/1043678242 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9800cd80 con 0x7f9ba80fff40 2024-03-17T23:57:38.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.861+0000 7f9ba4c00640 1 -- 172.21.15.12:0/1043678242 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b98010910 con 0x7f9ba80fff40 2024-03-17T23:57:38.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.861+0000 7f9bace00640 1 -- 172.21.15.12:0/1043678242 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba81a1af0 con 0x7f9ba80fff40 2024-03-17T23:57:38.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.861+0000 7f9ba4c00640 1 -- 172.21.15.12:0/1043678242 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98018990 con 0x7f9ba80fff40 2024-03-17T23:57:38.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.862+0000 7f9bace00640 1 -- 172.21.15.12:0/1043678242 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba81a2060 con 0x7f9ba80fff40 2024-03-17T23:57:38.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.862+0000 7f9bace00640 1 -- 172.21.15.12:0/1043678242 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba810ae80 con 0x7f9ba80fff40 2024-03-17T23:57:38.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.866+0000 7f9ba4c00640 1 -- 172.21.15.12:0/1043678242 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9b98010ac0 con 0x7f9ba80fff40 2024-03-17T23:57:38.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.867+0000 7f9ba4c00640 1 --2- 172.21.15.12:0/1043678242 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9b8c0a2880 0x7f9b8c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:38.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.868+0000 7f9ba4c00640 1 -- 172.21.15.12:0/1043678242 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f9b980cf240 con 0x7f9ba80fff40 2024-03-17T23:57:38.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.868+0000 7f9b9fe00640 1 --2- 172.21.15.12:0/1043678242 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9b8c0a2880 0x7f9b8c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:38.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.868+0000 7f9ba4c00640 1 -- 172.21.15.12:0/1043678242 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9b9809b8b0 con 0x7f9ba80fff40 2024-03-17T23:57:38.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:38.869+0000 7f9b9fe00640 1 --2- 172.21.15.12:0/1043678242 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9b8c0a2880 0x7f9b8c0a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9b9002c7c0 tx=0x7f9b90005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:39.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.071+0000 7f9bace00640 1 -- 172.21.15.12:0/1043678242 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9ba802d440 con 0x7f9ba80fff40 2024-03-17T23:57:39.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.072+0000 7f9ba4c00640 1 -- 172.21.15.12:0/1043678242 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f9b980cce20 con 0x7f9ba80fff40 2024-03-17T23:57:39.073 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-17T23:57:39.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.074+0000 7f9bace00640 1 -- 172.21.15.12:0/1043678242 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9b8c0a2880 msgr2=0x7f9b8c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:39.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.074+0000 7f9bace00640 1 --2- 172.21.15.12:0/1043678242 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9b8c0a2880 0x7f9b8c0a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9b9002c7c0 tx=0x7f9b90005fd0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.074+0000 7f9bace00640 1 -- 172.21.15.12:0/1043678242 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba80fff40 msgr2=0x7f9ba8199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:39.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.074+0000 7f9bace00640 1 --2- 172.21.15.12:0/1043678242 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba80fff40 0x7f9ba8199be0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800e9f0 tx=0x7f9b9800eeb0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.074+0000 7f9bace00640 1 -- 172.21.15.12:0/1043678242 shutdown_connections 2024-03-17T23:57:39.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.074+0000 7f9bace00640 1 --2- 172.21.15.12:0/1043678242 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f9b8c0a2880 0x7f9b8c0a4d40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.074+0000 7f9bace00640 1 --2- 172.21.15.12:0/1043678242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba8101240 0x7f9ba81a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.074+0000 7f9bace00640 1 --2- 172.21.15.12:0/1043678242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ba81008a0 0x7f9ba819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.074+0000 7f9bace00640 1 --2- 172.21.15.12:0/1043678242 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba80fff40 0x7f9ba8199be0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.074+0000 7f9bace00640 1 -- 172.21.15.12:0/1043678242 >> 172.21.15.12:0/1043678242 conn(0x7f9ba80fbb60 msgr2=0x7f9ba80fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:39.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.074+0000 7f9bace00640 1 -- 172.21.15.12:0/1043678242 shutdown_connections 2024-03-17T23:57:39.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.075+0000 7f9bace00640 1 -- 172.21.15.12:0/1043678242 wait complete. 2024-03-17T23:57:39.098 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:57:39.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:38 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1487652116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:57:39.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:38 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/38858923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:57:39.242 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:38 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1487652116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:57:39.242 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:38 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/38858923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:57:39.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:38 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1487652116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:57:39.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:38 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/38858923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:57:39.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.318+0000 7f2ea2800640 1 -- 172.21.15.12:0/3980880377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e9c100060 msgr2=0x7f2e9c102490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:39.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.318+0000 7f2ea2800640 1 --2- 172.21.15.12:0/3980880377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e9c100060 0x7f2e9c102490 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009ca0 tx=0x7f2e8c02c0f0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.318+0000 7f2ea2800640 1 -- 172.21.15.12:0/3980880377 shutdown_connections 2024-03-17T23:57:39.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.318+0000 7f2ea2800640 1 --2- 172.21.15.12:0/3980880377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e9c105380 0x7f2e9c105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.318+0000 7f2ea2800640 1 --2- 172.21.15.12:0/3980880377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e9c1029d0 0x7f2e9c104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.318+0000 7f2ea2800640 1 --2- 172.21.15.12:0/3980880377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e9c100060 0x7f2e9c102490 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.318+0000 7f2ea2800640 1 -- 172.21.15.12:0/3980880377 >> 172.21.15.12:0/3980880377 conn(0x7f2e9c0fbc80 msgr2=0x7f2e9c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:39.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.318+0000 7f2ea2800640 1 -- 172.21.15.12:0/3980880377 shutdown_connections 2024-03-17T23:57:39.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.318+0000 7f2ea2800640 1 -- 172.21.15.12:0/3980880377 wait complete. 2024-03-17T23:57:39.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.320+0000 7f2ea2800640 1 Processor -- start 2024-03-17T23:57:39.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.320+0000 7f2ea2800640 1 -- start start 2024-03-17T23:57:39.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.320+0000 7f2ea2800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e9c100060 0x7f2e9c19e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:39.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2ea2800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e9c1029d0 0x7f2e9c19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2ea2800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e9c105380 0x7f2e9c1a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2ea2800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c113a40 con 0x7f2e9c100060 2024-03-17T23:57:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2ea2800640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c1138c0 con 0x7f2e9c105380 2024-03-17T23:57:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2ea2800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c113bc0 con 0x7f2e9c1029d0 2024-03-17T23:57:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2e9be00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e9c105380 0x7f2e9c1a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2e9aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e9c1029d0 0x7f2e9c19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2e9b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e9c100060 0x7f2e9c19e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2e9be00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e9c105380 0x7f2e9c1a5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36318/0 (socket says 172.21.15.12:36318) 2024-03-17T23:57:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2e9be00640 1 -- 172.21.15.12:0/3422515743 learned_addr learned my addr 172.21.15.12:0/3422515743 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2e9be00640 1 -- 172.21.15.12:0/3422515743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e9c1029d0 msgr2=0x7f2e9c19e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2e9be00640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e9c1029d0 0x7f2e9c19e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2e9be00640 1 -- 172.21.15.12:0/3422515743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e9c100060 msgr2=0x7f2e9c19e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2e9be00640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e9c100060 0x7f2e9c19e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2e9be00640 1 -- 172.21.15.12:0/3422515743 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9c1a5cf0 con 0x7f2e9c105380 2024-03-17T23:57:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2e9aa00640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e9c1029d0 0x7f2e9c19e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.321+0000 7f2e9b400640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e9c100060 0x7f2e9c19e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.322+0000 7f2e9be00640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e9c105380 0x7f2e9c1a5680 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400e970 tx=0x7f2e9400ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.322+0000 7f2e93e00640 1 -- 172.21.15.12:0/3422515743 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9400ccc0 con 0x7f2e9c105380 2024-03-17T23:57:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.322+0000 7f2e93e00640 1 -- 172.21.15.12:0/3422515743 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e9400ce60 con 0x7f2e9c105380 2024-03-17T23:57:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.322+0000 7f2e93e00640 1 -- 172.21.15.12:0/3422515743 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e94010490 con 0x7f2e9c105380 2024-03-17T23:57:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.322+0000 7f2ea2800640 1 -- 172.21.15.12:0/3422515743 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e9c1a5fe0 con 0x7f2e9c105380 2024-03-17T23:57:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.322+0000 7f2ea2800640 1 -- 172.21.15.12:0/3422515743 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e9c1a64a0 con 0x7f2e9c105380 2024-03-17T23:57:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.323+0000 7f2ea2800640 1 -- 172.21.15.12:0/3422515743 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e9c0fd770 con 0x7f2e9c105380 2024-03-17T23:57:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.325+0000 7f2e93e00640 1 -- 172.21.15.12:0/3422515743 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2e94005020 con 0x7f2e9c105380 2024-03-17T23:57:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.327+0000 7f2e93e00640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2e800a2880 0x7f2e800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.328+0000 7f2e93e00640 1 -- 172.21.15.12:0/3422515743 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f2e940cf220 con 0x7f2e9c105380 2024-03-17T23:57:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.328+0000 7f2e9b400640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2e800a2880 0x7f2e800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.329+0000 7f2e9b400640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2e800a2880 0x7f2e800a4d40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c02c700 tx=0x7f2e8c005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.330+0000 7f2e93e00640 1 -- 172.21.15.12:0/3422515743 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2e9409bee0 con 0x7f2e9c105380 2024-03-17T23:57:39.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.536+0000 7f2ea2800640 1 -- 172.21.15.12:0/3422515743 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2e9c062480 con 0x7f2e9c105380 2024-03-17T23:57:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.537+0000 7f2e93e00640 1 -- 172.21.15.12:0/3422515743 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f2e9409b7e0 con 0x7f2e9c105380 2024-03-17T23:57:39.539 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-17T23:57:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.540+0000 7f2ea2800640 1 -- 172.21.15.12:0/3422515743 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2e800a2880 msgr2=0x7f2e800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.540+0000 7f2ea2800640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2e800a2880 0x7f2e800a4d40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c02c700 tx=0x7f2e8c005ef0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.540+0000 7f2ea2800640 1 -- 172.21.15.12:0/3422515743 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e9c105380 msgr2=0x7f2e9c1a5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.540+0000 7f2ea2800640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e9c105380 0x7f2e9c1a5680 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400e970 tx=0x7f2e9400ed70 comp rx=0 tx=0).stop 2024-03-17T23:57:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.540+0000 7f2ea2800640 1 -- 172.21.15.12:0/3422515743 shutdown_connections 2024-03-17T23:57:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.540+0000 7f2ea2800640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f2e800a2880 0x7f2e800a4d40 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.540+0000 7f2ea2800640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e9c105380 0x7f2e9c1a5680 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.540+0000 7f2ea2800640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e9c1029d0 0x7f2e9c19e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.540+0000 7f2ea2800640 1 --2- 172.21.15.12:0/3422515743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e9c100060 0x7f2e9c19e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.540+0000 7f2ea2800640 1 -- 172.21.15.12:0/3422515743 >> 172.21.15.12:0/3422515743 conn(0x7f2e9c0fbc80 msgr2=0x7f2e9c1032c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.540+0000 7f2ea2800640 1 -- 172.21.15.12:0/3422515743 shutdown_connections 2024-03-17T23:57:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.540+0000 7f2ea2800640 1 -- 172.21.15.12:0/3422515743 wait complete. 2024-03-17T23:57:39.563 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:57:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.766+0000 7fafdca00640 1 -- 172.21.15.12:0/662949792 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fafd81020d0 msgr2=0x7fafd81044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.766+0000 7fafdca00640 1 --2- 172.21.15.12:0/662949792 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fafd81020d0 0x7fafd81044c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009ca0 tx=0x7fafc802c0f0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.766+0000 7fafdca00640 1 -- 172.21.15.12:0/662949792 shutdown_connections 2024-03-17T23:57:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.766+0000 7fafdca00640 1 --2- 172.21.15.12:0/662949792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafd8107330 0x7fafd8109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.766+0000 7fafdca00640 1 --2- 172.21.15.12:0/662949792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd8104a00 0x7fafd8106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.766+0000 7fafdca00640 1 --2- 172.21.15.12:0/662949792 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fafd81020d0 0x7fafd81044c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.766+0000 7fafdca00640 1 -- 172.21.15.12:0/662949792 >> 172.21.15.12:0/662949792 conn(0x7fafd80fba60 msgr2=0x7fafd80fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.766+0000 7fafdca00640 1 -- 172.21.15.12:0/662949792 shutdown_connections 2024-03-17T23:57:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.766+0000 7fafdca00640 1 -- 172.21.15.12:0/662949792 wait complete. 2024-03-17T23:57:39.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.767+0000 7fafdca00640 1 Processor -- start 2024-03-17T23:57:39.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.768+0000 7fafdca00640 1 -- start start 2024-03-17T23:57:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.768+0000 7fafdca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafd81020d0 0x7fafd8199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.768+0000 7fafdca00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fafd8104a00 0x7fafd819a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.768+0000 7fafdca00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd8107330 0x7fafd81a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.768+0000 7fafdca00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fafd810d280 con 0x7fafd81020d0 2024-03-17T23:57:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.768+0000 7fafdca00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fafd810d100 con 0x7fafd8104a00 2024-03-17T23:57:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.768+0000 7fafdca00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fafd810d400 con 0x7fafd8107330 2024-03-17T23:57:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.768+0000 7fafd7400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafd81020d0 0x7fafd8199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.769+0000 7fafd6a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fafd8104a00 0x7fafd819a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.769+0000 7fafd7400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafd81020d0 0x7fafd8199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41882/0 (socket says 172.21.15.12:41882) 2024-03-17T23:57:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.769+0000 7fafd7400640 1 -- 172.21.15.12:0/1823665523 learned_addr learned my addr 172.21.15.12:0/1823665523 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.769+0000 7fafd7e00640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd8107330 0x7fafd81a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.769+0000 7fafd7400640 1 -- 172.21.15.12:0/1823665523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd8107330 msgr2=0x7fafd81a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.769+0000 7fafd7400640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd8107330 0x7fafd81a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.769+0000 7fafd7400640 1 -- 172.21.15.12:0/1823665523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fafd8104a00 msgr2=0x7fafd819a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.769+0000 7fafd7400640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fafd8104a00 0x7fafd819a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.769+0000 7fafd7400640 1 -- 172.21.15.12:0/1823665523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd81a1710 con 0x7fafd81020d0 2024-03-17T23:57:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.769+0000 7fafd6a00640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fafd8104a00 0x7fafd819a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.769+0000 7fafd7400640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafd81020d0 0x7fafd8199af0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fafc80325f0 tx=0x7fafc8032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.770+0000 7fafcfe00640 1 -- 172.21.15.12:0/1823665523 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8041020 con 0x7fafd81020d0 2024-03-17T23:57:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.770+0000 7fafcfe00640 1 -- 172.21.15.12:0/1823665523 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc8034cd0 con 0x7fafd81020d0 2024-03-17T23:57:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.770+0000 7fafcfe00640 1 -- 172.21.15.12:0/1823665523 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8039040 con 0x7fafd81020d0 2024-03-17T23:57:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.770+0000 7fafdca00640 1 -- 172.21.15.12:0/1823665523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd81a19a0 con 0x7fafd81020d0 2024-03-17T23:57:39.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.770+0000 7fafdca00640 1 -- 172.21.15.12:0/1823665523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd81a1eb0 con 0x7fafd81020d0 2024-03-17T23:57:39.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.771+0000 7fafdca00640 1 -- 172.21.15.12:0/1823665523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd8108e20 con 0x7fafd81020d0 2024-03-17T23:57:39.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.773+0000 7fafcfe00640 1 -- 172.21.15.12:0/1823665523 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fafc8044020 con 0x7fafd81020d0 2024-03-17T23:57:39.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.776+0000 7fafcfe00640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fafbc0a2880 0x7fafbc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:39.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.776+0000 7fafcfe00640 1 -- 172.21.15.12:0/1823665523 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fafc80f0f90 con 0x7fafd81020d0 2024-03-17T23:57:39.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.776+0000 7fafd6a00640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fafbc0a2880 0x7fafbc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:39.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.777+0000 7fafd6a00640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fafbc0a2880 0x7fafbc0a4d40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fafd819b000 tx=0x7fafc0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:39.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:39.779+0000 7fafcfe00640 1 -- 172.21.15.12:0/1823665523 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fafc804b020 con 0x7fafd81020d0 2024-03-17T23:57:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.026+0000 7fafdca00640 1 -- 172.21.15.12:0/1823665523 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafd802d1b0 con 0x7fafd81020d0 2024-03-17T23:57:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.027+0000 7fafcfe00640 1 -- 172.21.15.12:0/1823665523 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11936 (secure 0 0 0) 0x7fafc80b9070 con 0x7fafd81020d0 2024-03-17T23:57:40.029 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:40.030 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-17T23:54:56.312554+0000","modified":"2024-03-17T23:55:28.030481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24575,"mds_2":24530,"mds_3":24529,"mds_4":24514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.12:6841/2073607248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":2073607248},{"type":"v1","addr":"172.21.15.12:6841","nonce":2073607248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/2478742711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":2478742711},{"type":"v1","addr":"172.21.15.79:6839","nonce":2478742711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24575,24587,24530,24529,24514]},"id":1}]} 2024-03-17T23:57:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 28 2024-03-17T23:57:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.030+0000 7fafdca00640 1 -- 172.21.15.12:0/1823665523 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fafbc0a2880 msgr2=0x7fafbc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.030+0000 7fafdca00640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fafbc0a2880 0x7fafbc0a4d40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fafd819b000 tx=0x7fafc0005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.030+0000 7fafdca00640 1 -- 172.21.15.12:0/1823665523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafd81020d0 msgr2=0x7fafd8199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.030+0000 7fafdca00640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafd81020d0 0x7fafd8199af0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fafc80325f0 tx=0x7fafc8032ec0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.031+0000 7fafdca00640 1 -- 172.21.15.12:0/1823665523 shutdown_connections 2024-03-17T23:57:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.031+0000 7fafdca00640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fafbc0a2880 0x7fafbc0a4d40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.031+0000 7fafdca00640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd8107330 0x7fafd81a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.031+0000 7fafdca00640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fafd8104a00 0x7fafd819a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.031+0000 7fafdca00640 1 --2- 172.21.15.12:0/1823665523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafd81020d0 0x7fafd8199af0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.031+0000 7fafdca00640 1 -- 172.21.15.12:0/1823665523 >> 172.21.15.12:0/1823665523 conn(0x7fafd80fba60 msgr2=0x7fafd8101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.031+0000 7fafdca00640 1 -- 172.21.15.12:0/1823665523 shutdown_connections 2024-03-17T23:57:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.031+0000 7fafdca00640 1 -- 172.21.15.12:0/1823665523 wait complete. 2024-03-17T23:57:40.052 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:57:40.129 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:39 smithi012 ceph-mon[31541]: pgmap v406: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:40.130 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:39 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1043678242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:57:40.130 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:39 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3422515743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:57:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.274+0000 7f072d000640 1 -- 172.21.15.12:0/2913472485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0728104a00 msgr2=0x7f0728106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.274+0000 7f072d000640 1 --2- 172.21.15.12:0/2913472485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0728104a00 0x7f0728106df0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0714009900 tx=0x7f071402bc40 comp rx=0 tx=0).stop 2024-03-17T23:57:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.274+0000 7f072d000640 1 -- 172.21.15.12:0/2913472485 shutdown_connections 2024-03-17T23:57:40.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.274+0000 7f072d000640 1 --2- 172.21.15.12:0/2913472485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0728107330 0x7f0728109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.274+0000 7f072d000640 1 --2- 172.21.15.12:0/2913472485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0728104a00 0x7f0728106df0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.274+0000 7f072d000640 1 --2- 172.21.15.12:0/2913472485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f07281020d0 0x7f07281044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.274+0000 7f072d000640 1 -- 172.21.15.12:0/2913472485 >> 172.21.15.12:0/2913472485 conn(0x7f07280fba60 msgr2=0x7f07280fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:40.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.275+0000 7f072d000640 1 -- 172.21.15.12:0/2913472485 shutdown_connections 2024-03-17T23:57:40.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.275+0000 7f072d000640 1 -- 172.21.15.12:0/2913472485 wait complete. 2024-03-17T23:57:40.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.275+0000 7f072d000640 1 Processor -- start 2024-03-17T23:57:40.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.276+0000 7f072d000640 1 -- start start 2024-03-17T23:57:40.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.276+0000 7f072d000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07281020d0 0x7f0728199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:40.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.276+0000 7f072d000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0728104a00 0x7f072819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:40.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.276+0000 7f072d000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0728107330 0x7f07281a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:40.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.276+0000 7f072d000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f072810d2a0 con 0x7f0728104a00 2024-03-17T23:57:40.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.276+0000 7f072d000640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f072810d120 con 0x7f07281020d0 2024-03-17T23:57:40.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.276+0000 7f072d000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f072810d420 con 0x7f0728107330 2024-03-17T23:57:40.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.276+0000 7f0727e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0728107330 0x7f07281a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:40.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.276+0000 7f0726a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0728104a00 0x7f072819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:40.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.276+0000 7f0727400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07281020d0 0x7f0728199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:40.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.276+0000 7f0727e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0728107330 0x7f07281a1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:39666/0 (socket says 172.21.15.12:39666) 2024-03-17T23:57:40.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.276+0000 7f0727e00640 1 -- 172.21.15.12:0/2579465408 learned_addr learned my addr 172.21.15.12:0/2579465408 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:40.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.277+0000 7f0727400640 1 -- 172.21.15.12:0/2579465408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0728107330 msgr2=0x7f07281a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:40.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.277+0000 7f0727400640 1 --2- 172.21.15.12:0/2579465408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0728107330 0x7f07281a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.277+0000 7f0727400640 1 -- 172.21.15.12:0/2579465408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0728104a00 msgr2=0x7f072819a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:40.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.277+0000 7f0727400640 1 --2- 172.21.15.12:0/2579465408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0728104a00 0x7f072819a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.277+0000 7f0727400640 1 -- 172.21.15.12:0/2579465408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07281a1770 con 0x7f07281020d0 2024-03-17T23:57:40.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.277+0000 7f0727400640 1 --2- 172.21.15.12:0/2579465408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07281020d0 0x7f0728199b50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f071800eb00 tx=0x7f071800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:40.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.277+0000 7f071fe00640 1 -- 172.21.15.12:0/2579465408 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0718010430 con 0x7f07281020d0 2024-03-17T23:57:40.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.277+0000 7f071fe00640 1 -- 172.21.15.12:0/2579465408 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0718010ab0 con 0x7f07281020d0 2024-03-17T23:57:40.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.277+0000 7f071fe00640 1 -- 172.21.15.12:0/2579465408 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0718017bb0 con 0x7f07281020d0 2024-03-17T23:57:40.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.277+0000 7f072d000640 1 -- 172.21.15.12:0/2579465408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07281a1a60 con 0x7f07281020d0 2024-03-17T23:57:40.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.277+0000 7f072d000640 1 -- 172.21.15.12:0/2579465408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07281a1f30 con 0x7f07281020d0 2024-03-17T23:57:40.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.278+0000 7f072d000640 1 -- 172.21.15.12:0/2579465408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0728108e20 con 0x7f07281020d0 2024-03-17T23:57:40.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.280+0000 7f071fe00640 1 -- 172.21.15.12:0/2579465408 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0718010c60 con 0x7f07281020d0 2024-03-17T23:57:40.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.282+0000 7f071fe00640 1 --2- 172.21.15.12:0/2579465408 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f070c0a2880 0x7f070c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:40.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.282+0000 7f071fe00640 1 -- 172.21.15.12:0/2579465408 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f07180cf710 con 0x7f07281020d0 2024-03-17T23:57:40.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.282+0000 7f0726a00640 1 --2- 172.21.15.12:0/2579465408 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f070c0a2880 0x7f070c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:40.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.283+0000 7f0726a00640 1 --2- 172.21.15.12:0/2579465408 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f070c0a2880 0x7f070c0a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f071402c180 tx=0x7f071402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:40.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.285+0000 7f071fe00640 1 -- 172.21.15.12:0/2579465408 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f071809c3d0 con 0x7f07281020d0 2024-03-17T23:57:40.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:39 smithi079 ceph-mon[36241]: pgmap v406: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:40.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:39 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1043678242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:57:40.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:39 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3422515743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:57:40.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:39 smithi191 ceph-mon[36708]: pgmap v406: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:40.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:39 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1043678242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:57:40.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:39 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3422515743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:57:40.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.492+0000 7f072d000640 1 -- 172.21.15.12:0/2579465408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f07280623e0 con 0x7f07281020d0 2024-03-17T23:57:40.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.494+0000 7f071fe00640 1 -- 172.21.15.12:0/2579465408 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24618 (secure 0 0 0) 0x7f071809bcd0 con 0x7f07281020d0 2024-03-17T23:57:40.496 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:40.497 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":94,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:56:55.719277+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:57:40.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.497+0000 7f072d000640 1 -- 172.21.15.12:0/2579465408 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f070c0a2880 msgr2=0x7f070c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:40.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.497+0000 7f072d000640 1 --2- 172.21.15.12:0/2579465408 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f070c0a2880 0x7f070c0a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f071402c180 tx=0x7f071402cf90 comp rx=0 tx=0).stop 2024-03-17T23:57:40.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.497+0000 7f072d000640 1 -- 172.21.15.12:0/2579465408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07281020d0 msgr2=0x7f0728199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:40.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.497+0000 7f072d000640 1 --2- 172.21.15.12:0/2579465408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07281020d0 0x7f0728199b50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f071800eb00 tx=0x7f071800c2d0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.497+0000 7f072d000640 1 -- 172.21.15.12:0/2579465408 shutdown_connections 2024-03-17T23:57:40.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.497+0000 7f072d000640 1 --2- 172.21.15.12:0/2579465408 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f070c0a2880 0x7f070c0a4d40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.497+0000 7f072d000640 1 --2- 172.21.15.12:0/2579465408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0728107330 0x7f07281a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.497+0000 7f072d000640 1 --2- 172.21.15.12:0/2579465408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0728104a00 0x7f072819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.497+0000 7f072d000640 1 --2- 172.21.15.12:0/2579465408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07281020d0 0x7f0728199b50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.497+0000 7f072d000640 1 -- 172.21.15.12:0/2579465408 >> 172.21.15.12:0/2579465408 conn(0x7f07280fba60 msgr2=0x7f07281017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:40.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.497+0000 7f072d000640 1 -- 172.21.15.12:0/2579465408 shutdown_connections 2024-03-17T23:57:40.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.497+0000 7f072d000640 1 -- 172.21.15.12:0/2579465408 wait complete. 2024-03-17T23:57:40.518 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:49:38.640282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:54:54.540345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:54:55.803926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:55:00.355066+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:57:40.519 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:57:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.738+0000 7f2016e00640 1 -- 172.21.15.12:0/600125165 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2010104bf0 msgr2=0x7f2010105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.738+0000 7f2016e00640 1 --2- 172.21.15.12:0/600125165 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2010104bf0 0x7f2010105070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009900 tx=0x7f1ffc02bc50 comp rx=0 tx=0).stop 2024-03-17T23:57:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.739+0000 7f2016e00640 1 -- 172.21.15.12:0/600125165 shutdown_connections 2024-03-17T23:57:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.739+0000 7f2016e00640 1 --2- 172.21.15.12:0/600125165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20101055b0 0x7f201010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.739+0000 7f2016e00640 1 --2- 172.21.15.12:0/600125165 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2010104bf0 0x7f2010105070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.739+0000 7f2016e00640 1 --2- 172.21.15.12:0/600125165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2010103a00 0x7f2010103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.739+0000 7f2016e00640 1 -- 172.21.15.12:0/600125165 >> 172.21.15.12:0/600125165 conn(0x7f20100fef90 msgr2=0x7f20101013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.739+0000 7f2016e00640 1 -- 172.21.15.12:0/600125165 shutdown_connections 2024-03-17T23:57:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.739+0000 7f2016e00640 1 -- 172.21.15.12:0/600125165 wait complete. 2024-03-17T23:57:40.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.740+0000 7f2016e00640 1 Processor -- start 2024-03-17T23:57:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.740+0000 7f2016e00640 1 -- start start 2024-03-17T23:57:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.740+0000 7f2016e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2010103a00 0x7f2010199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.740+0000 7f2016e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2010104bf0 0x7f201019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.740+0000 7f2016e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20101055b0 0x7f20101a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.740+0000 7f2016e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f201010f6b0 con 0x7f20101055b0 2024-03-17T23:57:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.740+0000 7f2016e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f201010f530 con 0x7f2010103a00 2024-03-17T23:57:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.740+0000 7f2016e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f201010f830 con 0x7f2010104bf0 2024-03-17T23:57:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.741+0000 7f200fe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2010103a00 0x7f2010199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.741+0000 7f200f400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2010104bf0 0x7f201019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.741+0000 7f200fe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2010103a00 0x7f2010199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36382/0 (socket says 172.21.15.12:36382) 2024-03-17T23:57:40.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.741+0000 7f200fe00640 1 -- 172.21.15.12:0/1791652680 learned_addr learned my addr 172.21.15.12:0/1791652680 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:40.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.741+0000 7f2014e00640 1 --2- 172.21.15.12:0/1791652680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20101055b0 0x7f20101a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:40.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.741+0000 7f200fe00640 1 -- 172.21.15.12:0/1791652680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2010104bf0 msgr2=0x7f201019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:40.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.741+0000 7f200fe00640 1 --2- 172.21.15.12:0/1791652680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2010104bf0 0x7f201019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.741+0000 7f200fe00640 1 -- 172.21.15.12:0/1791652680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20101055b0 msgr2=0x7f20101a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:40.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.741+0000 7f200fe00640 1 --2- 172.21.15.12:0/1791652680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20101055b0 0x7f20101a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.741+0000 7f200fe00640 1 -- 172.21.15.12:0/1791652680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20101a1960 con 0x7f2010103a00 2024-03-17T23:57:40.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.742+0000 7f200fe00640 1 --2- 172.21.15.12:0/1791652680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2010103a00 0x7f2010199d50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f200000d950 tx=0x7f200000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:40.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.742+0000 7f200cc00640 1 -- 172.21.15.12:0/1791652680 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000017020 con 0x7f2010103a00 2024-03-17T23:57:40.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.742+0000 7f200cc00640 1 -- 172.21.15.12:0/1791652680 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2000009c30 con 0x7f2010103a00 2024-03-17T23:57:40.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.742+0000 7f200cc00640 1 -- 172.21.15.12:0/1791652680 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200000f720 con 0x7f2010103a00 2024-03-17T23:57:40.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.742+0000 7f2016e00640 1 -- 172.21.15.12:0/1791652680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20101a1c50 con 0x7f2010103a00 2024-03-17T23:57:40.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.742+0000 7f2016e00640 1 -- 172.21.15.12:0/1791652680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20101a2150 con 0x7f2010103a00 2024-03-17T23:57:40.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.743+0000 7f2016e00640 1 -- 172.21.15.12:0/1791652680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f201010ad40 con 0x7f2010103a00 2024-03-17T23:57:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.747+0000 7f200cc00640 1 -- 172.21.15.12:0/1791652680 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f200001a020 con 0x7f2010103a00 2024-03-17T23:57:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.748+0000 7f200cc00640 1 --2- 172.21.15.12:0/1791652680 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1ff40a2880 0x7f1ff40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.748+0000 7f200cc00640 1 -- 172.21.15.12:0/1791652680 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f20000ced10 con 0x7f2010103a00 2024-03-17T23:57:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.748+0000 7f200f400640 1 --2- 172.21.15.12:0/1791652680 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1ff40a2880 0x7f1ff40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.749+0000 7f200f400640 1 --2- 172.21.15.12:0/1791652680 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1ff40a2880 0x7f1ff40a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc02c260 tx=0x7f1ffc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.752+0000 7f200cc00640 1 -- 172.21.15.12:0/1791652680 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f20000d4020 con 0x7f2010103a00 2024-03-17T23:57:40.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.961+0000 7f2016e00640 1 -- 172.21.15.12:0/1791652680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2010062520 con 0x7f2010103a00 2024-03-17T23:57:40.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.962+0000 7f200cc00640 1 -- 172.21.15.12:0/1791652680 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f2000021020 con 0x7f2010103a00 2024-03-17T23:57:40.964 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-17T23:57:40.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.965+0000 7f2016e00640 1 -- 172.21.15.12:0/1791652680 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1ff40a2880 msgr2=0x7f1ff40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:40.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.965+0000 7f2016e00640 1 --2- 172.21.15.12:0/1791652680 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1ff40a2880 0x7f1ff40a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc02c260 tx=0x7f1ffc000bc0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.965+0000 7f2016e00640 1 -- 172.21.15.12:0/1791652680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2010103a00 msgr2=0x7f2010199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:40.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.965+0000 7f2016e00640 1 --2- 172.21.15.12:0/1791652680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2010103a00 0x7f2010199d50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f200000d950 tx=0x7f200000de10 comp rx=0 tx=0).stop 2024-03-17T23:57:40.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.965+0000 7f2016e00640 1 -- 172.21.15.12:0/1791652680 shutdown_connections 2024-03-17T23:57:40.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.965+0000 7f2016e00640 1 --2- 172.21.15.12:0/1791652680 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1ff40a2880 0x7f1ff40a4d40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.965+0000 7f2016e00640 1 --2- 172.21.15.12:0/1791652680 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20101055b0 0x7f20101a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.965+0000 7f2016e00640 1 --2- 172.21.15.12:0/1791652680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2010104bf0 0x7f201019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.965+0000 7f2016e00640 1 --2- 172.21.15.12:0/1791652680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2010103a00 0x7f2010199d50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:40.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.965+0000 7f2016e00640 1 -- 172.21.15.12:0/1791652680 >> 172.21.15.12:0/1791652680 conn(0x7f20100fef90 msgr2=0x7f2010107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:40.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.965+0000 7f2016e00640 1 -- 172.21.15.12:0/1791652680 shutdown_connections 2024-03-17T23:57:40.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:40.965+0000 7f2016e00640 1 -- 172.21.15.12:0/1791652680 wait complete. 2024-03-17T23:57:40.990 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:57:41.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:40 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1823665523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:41.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:40 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2579465408' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:41.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.222+0000 7f5351e00640 1 -- 172.21.15.12:0/3566759681 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f534c1054b0 msgr2=0x7f534c10bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:41.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.222+0000 7f5351e00640 1 --2- 172.21.15.12:0/3566759681 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f534c1054b0 0x7f534c10bd40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5344009ca0 tx=0x7f534402c100 comp rx=0 tx=0).stop 2024-03-17T23:57:41.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.222+0000 7f5351e00640 1 -- 172.21.15.12:0/3566759681 shutdown_connections 2024-03-17T23:57:41.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.222+0000 7f5351e00640 1 --2- 172.21.15.12:0/3566759681 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f534c1054b0 0x7f534c10bd40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.222+0000 7f5351e00640 1 --2- 172.21.15.12:0/3566759681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f534c104af0 0x7f534c104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.222+0000 7f5351e00640 1 --2- 172.21.15.12:0/3566759681 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f534c103900 0x7f534c103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.222+0000 7f5351e00640 1 -- 172.21.15.12:0/3566759681 >> 172.21.15.12:0/3566759681 conn(0x7f534c0fee90 msgr2=0x7f534c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:41.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.222+0000 7f5351e00640 1 -- 172.21.15.12:0/3566759681 shutdown_connections 2024-03-17T23:57:41.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.223+0000 7f5351e00640 1 -- 172.21.15.12:0/3566759681 wait complete. 2024-03-17T23:57:41.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.224+0000 7f5351e00640 1 Processor -- start 2024-03-17T23:57:41.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.224+0000 7f5351e00640 1 -- start start 2024-03-17T23:57:41.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.224+0000 7f5351e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f534c103900 0x7f534c199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.224+0000 7f5351e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f534c104af0 0x7f534c19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.224+0000 7f5351e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f534c1054b0 0x7f534c1a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.224+0000 7f5351e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f534c10f560 con 0x7f534c103900 2024-03-17T23:57:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.224+0000 7f5351e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f534c10f3e0 con 0x7f534c1054b0 2024-03-17T23:57:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.224+0000 7f5351e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f534c10f6e0 con 0x7f534c104af0 2024-03-17T23:57:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.225+0000 7f534be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f534c104af0 0x7f534c19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.225+0000 7f534be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f534c104af0 0x7f534c19a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:39690/0 (socket says 172.21.15.12:39690) 2024-03-17T23:57:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.225+0000 7f534be00640 1 -- 172.21.15.12:0/1379973833 learned_addr learned my addr 172.21.15.12:0/1379973833 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.225+0000 7f5351400640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f534c1054b0 0x7f534c1a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.225+0000 7f5350a00640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f534c103900 0x7f534c199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.225+0000 7f5351400640 1 -- 172.21.15.12:0/1379973833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f534c104af0 msgr2=0x7f534c19a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.225+0000 7f5351400640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f534c104af0 0x7f534c19a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.225+0000 7f5351400640 1 -- 172.21.15.12:0/1379973833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f534c103900 msgr2=0x7f534c199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.225+0000 7f5351400640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f534c103900 0x7f534c199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.225+0000 7f5351400640 1 -- 172.21.15.12:0/1379973833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534c1a17a0 con 0x7f534c1054b0 2024-03-17T23:57:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.225+0000 7f5351400640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f534c1054b0 0x7f534c1a1130 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5344037040 tx=0x7f5344035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.226+0000 7f5349600640 1 -- 172.21.15.12:0/1379973833 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5344039040 con 0x7f534c1054b0 2024-03-17T23:57:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.226+0000 7f5349600640 1 -- 172.21.15.12:0/1379973833 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5344044070 con 0x7f534c1054b0 2024-03-17T23:57:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.226+0000 7f5349600640 1 -- 172.21.15.12:0/1379973833 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5344004350 con 0x7f534c1054b0 2024-03-17T23:57:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.226+0000 7f5351e00640 1 -- 172.21.15.12:0/1379973833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534c1a1a30 con 0x7f534c1054b0 2024-03-17T23:57:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.226+0000 7f5351e00640 1 -- 172.21.15.12:0/1379973833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f534c1a1f40 con 0x7f534c1054b0 2024-03-17T23:57:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.227+0000 7f5351e00640 1 -- 172.21.15.12:0/1379973833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f534c10ac20 con 0x7f534c1054b0 2024-03-17T23:57:41.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.229+0000 7f5349600640 1 -- 172.21.15.12:0/1379973833 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f53440044f0 con 0x7f534c1054b0 2024-03-17T23:57:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.230+0000 7f5349600640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f53340a2880 0x7f53340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.230+0000 7f5349600640 1 -- 172.21.15.12:0/1379973833 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f53440f1380 con 0x7f534c1054b0 2024-03-17T23:57:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.230+0000 7f5350a00640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f53340a2880 0x7f53340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.231+0000 7f5350a00640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f53340a2880 0x7f53340a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f534c104950 tx=0x7f5340005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.234+0000 7f5349600640 1 -- 172.21.15.12:0/1379973833 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f53440be040 con 0x7f534c1054b0 2024-03-17T23:57:41.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:40 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1823665523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:41.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:40 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2579465408' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:41.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:40 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1823665523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:41.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:40 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2579465408' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.446+0000 7f5351e00640 1 -- 172.21.15.12:0/1379973833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f534c0623e0 con 0x7f534c1054b0 2024-03-17T23:57:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.447+0000 7f5349600640 1 -- 172.21.15.12:0/1379973833 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f53440098b0 con 0x7f534c1054b0 2024-03-17T23:57:41.448 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-17T23:57:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.449+0000 7f5351e00640 1 -- 172.21.15.12:0/1379973833 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f53340a2880 msgr2=0x7f53340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.449+0000 7f5351e00640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f53340a2880 0x7f53340a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f534c104950 tx=0x7f5340005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.449+0000 7f5351e00640 1 -- 172.21.15.12:0/1379973833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f534c1054b0 msgr2=0x7f534c1a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.449+0000 7f5351e00640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f534c1054b0 0x7f534c1a1130 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5344037040 tx=0x7f5344035d10 comp rx=0 tx=0).stop 2024-03-17T23:57:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.450+0000 7f5351e00640 1 -- 172.21.15.12:0/1379973833 shutdown_connections 2024-03-17T23:57:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.450+0000 7f5351e00640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f53340a2880 0x7f53340a4d40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.450+0000 7f5351e00640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f534c1054b0 0x7f534c1a1130 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.450+0000 7f5351e00640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f534c104af0 0x7f534c19a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.450+0000 7f5351e00640 1 --2- 172.21.15.12:0/1379973833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f534c103900 0x7f534c199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.450+0000 7f5351e00640 1 -- 172.21.15.12:0/1379973833 >> 172.21.15.12:0/1379973833 conn(0x7f534c0fee90 msgr2=0x7f534c107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.450+0000 7f5351e00640 1 -- 172.21.15.12:0/1379973833 shutdown_connections 2024-03-17T23:57:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.450+0000 7f5351e00640 1 -- 172.21.15.12:0/1379973833 wait complete. 2024-03-17T23:57:41.473 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:57:41.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.696+0000 7f6c67200640 1 -- 172.21.15.12:0/2925112559 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c68107460 msgr2=0x7f6c68109850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:41.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.696+0000 7f6c67200640 1 --2- 172.21.15.12:0/2925112559 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c68107460 0x7f6c68109850 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009ca0 tx=0x7f6c6002c100 comp rx=0 tx=0).stop 2024-03-17T23:57:41.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.696+0000 7f6c67200640 1 -- 172.21.15.12:0/2925112559 shutdown_connections 2024-03-17T23:57:41.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.696+0000 7f6c67200640 1 --2- 172.21.15.12:0/2925112559 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c68107460 0x7f6c68109850 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.696+0000 7f6c67200640 1 --2- 172.21.15.12:0/2925112559 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c68104b30 0x7f6c68106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.696+0000 7f6c67200640 1 --2- 172.21.15.12:0/2925112559 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c68102200 0x7f6c681045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.696+0000 7f6c67200640 1 -- 172.21.15.12:0/2925112559 >> 172.21.15.12:0/2925112559 conn(0x7f6c680fbb50 msgr2=0x7f6c680fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:41.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.696+0000 7f6c67200640 1 -- 172.21.15.12:0/2925112559 shutdown_connections 2024-03-17T23:57:41.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.696+0000 7f6c67200640 1 -- 172.21.15.12:0/2925112559 wait complete. 2024-03-17T23:57:41.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.697+0000 7f6c67200640 1 Processor -- start 2024-03-17T23:57:41.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.697+0000 7f6c67200640 1 -- start start 2024-03-17T23:57:41.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.697+0000 7f6c67200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c68102200 0x7f6c68199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:41.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.697+0000 7f6c67200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c68104b30 0x7f6c6819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:41.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.697+0000 7f6c67200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c68107460 0x7f6c681a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:41.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.697+0000 7f6c67200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6810d350 con 0x7f6c68104b30 2024-03-17T23:57:41.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.697+0000 7f6c67200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6810d1d0 con 0x7f6c68107460 2024-03-17T23:57:41.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.697+0000 7f6c67200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6810d4d0 con 0x7f6c68102200 2024-03-17T23:57:41.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.698+0000 7f6c65e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c68102200 0x7f6c68199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:41.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.698+0000 7f6c65e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c68102200 0x7f6c68199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:39710/0 (socket says 172.21.15.12:39710) 2024-03-17T23:57:41.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.698+0000 7f6c65e00640 1 -- 172.21.15.12:0/570915636 learned_addr learned my addr 172.21.15.12:0/570915636 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:41.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.698+0000 7f6c5fe00640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c68104b30 0x7f6c6819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:41.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.698+0000 7f6c66800640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c68107460 0x7f6c681a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:41.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.698+0000 7f6c65e00640 1 -- 172.21.15.12:0/570915636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c68107460 msgr2=0x7f6c681a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:41.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.698+0000 7f6c65e00640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c68107460 0x7f6c681a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.698+0000 7f6c65e00640 1 -- 172.21.15.12:0/570915636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c68104b30 msgr2=0x7f6c6819a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:41.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.698+0000 7f6c65e00640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c68104b30 0x7f6c6819a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.698+0000 7f6c65e00640 1 -- 172.21.15.12:0/570915636 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c681a17c0 con 0x7f6c68102200 2024-03-17T23:57:41.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.698+0000 7f6c66800640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c68107460 0x7f6c681a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:41.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.699+0000 7f6c65e00640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c68102200 0x7f6c68199ba0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6c5800e9f0 tx=0x7f6c5800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.699+0000 7f6c5ea00640 1 -- 172.21.15.12:0/570915636 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5800cd80 con 0x7f6c68102200 2024-03-17T23:57:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.699+0000 7f6c5ea00640 1 -- 172.21.15.12:0/570915636 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c58010910 con 0x7f6c68102200 2024-03-17T23:57:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.699+0000 7f6c5ea00640 1 -- 172.21.15.12:0/570915636 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58018990 con 0x7f6c68102200 2024-03-17T23:57:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.699+0000 7f6c67200640 1 -- 172.21.15.12:0/570915636 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c681a1ab0 con 0x7f6c68102200 2024-03-17T23:57:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.699+0000 7f6c67200640 1 -- 172.21.15.12:0/570915636 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c681a2000 con 0x7f6c68102200 2024-03-17T23:57:41.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.700+0000 7f6c67200640 1 -- 172.21.15.12:0/570915636 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c68108f50 con 0x7f6c68102200 2024-03-17T23:57:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.702+0000 7f6c5ea00640 1 -- 172.21.15.12:0/570915636 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6c58010ac0 con 0x7f6c68102200 2024-03-17T23:57:41.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.703+0000 7f6c5ea00640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6c4c0a2800 0x7f6c4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:41.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.703+0000 7f6c5ea00640 1 -- 172.21.15.12:0/570915636 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6c580cf190 con 0x7f6c68102200 2024-03-17T23:57:41.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.704+0000 7f6c5fe00640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6c4c0a2800 0x7f6c4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:41.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.704+0000 7f6c5fe00640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6c4c0a2800 0x7f6c4c0a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6c6819b0b0 tx=0x7f6c50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:41.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.712+0000 7f6c5ea00640 1 -- 172.21.15.12:0/570915636 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6c5809b800 con 0x7f6c68102200 2024-03-17T23:57:41.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.930+0000 7f6c67200640 1 -- 172.21.15.12:0/570915636 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6c680623e0 con 0x7f6c68102200 2024-03-17T23:57:41.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.931+0000 7f6c5ea00640 1 -- 172.21.15.12:0/570915636 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f6c5809f8a0 con 0x7f6c68102200 2024-03-17T23:57:41.932 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-17T23:57:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.933+0000 7f6c67200640 1 -- 172.21.15.12:0/570915636 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6c4c0a2800 msgr2=0x7f6c4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.933+0000 7f6c67200640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6c4c0a2800 0x7f6c4c0a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6c6819b0b0 tx=0x7f6c50005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.933+0000 7f6c67200640 1 -- 172.21.15.12:0/570915636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c68102200 msgr2=0x7f6c68199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.933+0000 7f6c67200640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c68102200 0x7f6c68199ba0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6c5800e9f0 tx=0x7f6c5800eeb0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.933+0000 7f6c67200640 1 -- 172.21.15.12:0/570915636 shutdown_connections 2024-03-17T23:57:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.933+0000 7f6c67200640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6c4c0a2800 0x7f6c4c0a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.933+0000 7f6c67200640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c68107460 0x7f6c681a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.933+0000 7f6c67200640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c68104b30 0x7f6c6819a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.933+0000 7f6c67200640 1 --2- 172.21.15.12:0/570915636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c68102200 0x7f6c68199ba0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.933+0000 7f6c67200640 1 -- 172.21.15.12:0/570915636 >> 172.21.15.12:0/570915636 conn(0x7f6c680fbb50 msgr2=0x7f6c68101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.934+0000 7f6c67200640 1 -- 172.21.15.12:0/570915636 shutdown_connections 2024-03-17T23:57:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:41.934+0000 7f6c67200640 1 -- 172.21.15.12:0/570915636 wait complete. 2024-03-17T23:57:41.954 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:57:42.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:41 smithi012 ceph-mon[31541]: pgmap v407: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:42.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:41 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1791652680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:57:42.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:41 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1379973833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:57:42.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.171+0000 7f490a600640 1 -- 172.21.15.12:0/3425728634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4904075820 msgr2=0x7f4904073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:42.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.171+0000 7f490a600640 1 --2- 172.21.15.12:0/3425728634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4904075820 0x7f4904073c70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009900 tx=0x7f48f802bc40 comp rx=0 tx=0).stop 2024-03-17T23:57:42.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.171+0000 7f490a600640 1 -- 172.21.15.12:0/3425728634 shutdown_connections 2024-03-17T23:57:42.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.171+0000 7f490a600640 1 --2- 172.21.15.12:0/3425728634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4904109580 0x7f490410ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.171+0000 7f490a600640 1 --2- 172.21.15.12:0/3425728634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4904074240 0x7f49040746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.171+0000 7f490a600640 1 --2- 172.21.15.12:0/3425728634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4904075820 0x7f4904073c70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.171+0000 7f490a600640 1 -- 172.21.15.12:0/3425728634 >> 172.21.15.12:0/3425728634 conn(0x7f49040fcc50 msgr2=0x7f49040ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:42.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.171+0000 7f490a600640 1 -- 172.21.15.12:0/3425728634 shutdown_connections 2024-03-17T23:57:42.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.171+0000 7f490a600640 1 -- 172.21.15.12:0/3425728634 wait complete. 2024-03-17T23:57:42.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.172+0000 7f490a600640 1 Processor -- start 2024-03-17T23:57:42.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.173+0000 7f490a600640 1 -- start start 2024-03-17T23:57:42.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.173+0000 7f490a600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4904074240 0x7f4904199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:42.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.173+0000 7f490a600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4904075820 0x7f490419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:42.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.173+0000 7f490a600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4904109580 0x7f49041a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:42.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.173+0000 7f490a600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f490410f570 con 0x7f4904074240 2024-03-17T23:57:42.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.173+0000 7f490a600640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f490410f3f0 con 0x7f4904075820 2024-03-17T23:57:42.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.173+0000 7f490a600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f490410f6f0 con 0x7f4904109580 2024-03-17T23:57:42.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.173+0000 7f4903e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4904075820 0x7f490419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:42.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.173+0000 7f4903e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4904075820 0x7f490419a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36436/0 (socket says 172.21.15.12:36436) 2024-03-17T23:57:42.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.173+0000 7f4903e00640 1 -- 172.21.15.12:0/2580155406 learned_addr learned my addr 172.21.15.12:0/2580155406 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:42.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.173+0000 7f4909c00640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4904109580 0x7f49041a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:42.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.173+0000 7f4909200640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4904074240 0x7f4904199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:42.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.173+0000 7f4903e00640 1 -- 172.21.15.12:0/2580155406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4904109580 msgr2=0x7f49041a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:42.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.174+0000 7f4903e00640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4904109580 0x7f49041a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.174+0000 7f4903e00640 1 -- 172.21.15.12:0/2580155406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4904074240 msgr2=0x7f4904199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:42.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.174+0000 7f4903e00640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4904074240 0x7f4904199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.174+0000 7f4903e00640 1 -- 172.21.15.12:0/2580155406 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49041a17a0 con 0x7f4904075820 2024-03-17T23:57:42.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.174+0000 7f4903e00640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4904075820 0x7f490419a0c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f48f000ea60 tx=0x7f48f000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:42.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.174+0000 7f4901600640 1 -- 172.21.15.12:0/2580155406 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f000cde0 con 0x7f4904075820 2024-03-17T23:57:42.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.174+0000 7f4901600640 1 -- 172.21.15.12:0/2580155406 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48f00035f0 con 0x7f4904075820 2024-03-17T23:57:42.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.174+0000 7f490a600640 1 -- 172.21.15.12:0/2580155406 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49041a1a90 con 0x7f4904075820 2024-03-17T23:57:42.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.174+0000 7f4901600640 1 -- 172.21.15.12:0/2580155406 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f0010670 con 0x7f4904075820 2024-03-17T23:57:42.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.174+0000 7f490a600640 1 -- 172.21.15.12:0/2580155406 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49041a2030 con 0x7f4904075820 2024-03-17T23:57:42.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.175+0000 7f490a600640 1 -- 172.21.15.12:0/2580155406 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49041082b0 con 0x7f4904075820 2024-03-17T23:57:42.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.178+0000 7f4901600640 1 -- 172.21.15.12:0/2580155406 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f48f00037a0 con 0x7f4904075820 2024-03-17T23:57:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.179+0000 7f4901600640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f48ec0a2880 0x7f48ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.179+0000 7f4901600640 1 -- 172.21.15.12:0/2580155406 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f48f00cf080 con 0x7f4904075820 2024-03-17T23:57:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.180+0000 7f4909200640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f48ec0a2880 0x7f48ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:42.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.180+0000 7f4909200640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f48ec0a2880 0x7f48ec0a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f48f802c250 tx=0x7f48f802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:42.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.183+0000 7f4901600640 1 -- 172.21.15.12:0/2580155406 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f48f009bd40 con 0x7f4904075820 2024-03-17T23:57:42.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:41 smithi079 ceph-mon[36241]: pgmap v407: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:42.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:41 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1791652680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:57:42.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:41 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1379973833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:57:42.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:41 smithi191 ceph-mon[36708]: pgmap v407: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:42.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:41 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1791652680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-17T23:57:42.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:41 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1379973833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-17T23:57:42.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.388+0000 7f490a600640 1 -- 172.21.15.12:0/2580155406 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f49040623e0 con 0x7f4904075820 2024-03-17T23:57:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.388+0000 7f4901600640 1 -- 172.21.15.12:0/2580155406 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f48f0097030 con 0x7f4904075820 2024-03-17T23:57:42.390 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-17T23:57:42.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.391+0000 7f490a600640 1 -- 172.21.15.12:0/2580155406 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f48ec0a2880 msgr2=0x7f48ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.391+0000 7f490a600640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f48ec0a2880 0x7f48ec0a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f48f802c250 tx=0x7f48f802cf90 comp rx=0 tx=0).stop 2024-03-17T23:57:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.391+0000 7f490a600640 1 -- 172.21.15.12:0/2580155406 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4904075820 msgr2=0x7f490419a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.391+0000 7f490a600640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4904075820 0x7f490419a0c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f48f000ea60 tx=0x7f48f000ef20 comp rx=0 tx=0).stop 2024-03-17T23:57:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.391+0000 7f490a600640 1 -- 172.21.15.12:0/2580155406 shutdown_connections 2024-03-17T23:57:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.391+0000 7f490a600640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f48ec0a2880 0x7f48ec0a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.391+0000 7f490a600640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4904109580 0x7f49041a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.391+0000 7f490a600640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4904075820 0x7f490419a0c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.391+0000 7f490a600640 1 --2- 172.21.15.12:0/2580155406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4904074240 0x7f4904199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.391+0000 7f490a600640 1 -- 172.21.15.12:0/2580155406 >> 172.21.15.12:0/2580155406 conn(0x7f49040fcc50 msgr2=0x7f4904105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.391+0000 7f490a600640 1 -- 172.21.15.12:0/2580155406 shutdown_connections 2024-03-17T23:57:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.391+0000 7f490a600640 1 -- 172.21.15.12:0/2580155406 wait complete. 2024-03-17T23:57:42.413 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:57:42.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.622+0000 7fa492600640 1 -- 172.21.15.12:0/3637655290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa48c102820 msgr2=0x7fa48c102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:42.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.622+0000 7fa492600640 1 --2- 172.21.15.12:0/3637655290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa48c102820 0x7fa48c102ca0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa478009900 tx=0x7fa47802bc40 comp rx=0 tx=0).stop 2024-03-17T23:57:42.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.622+0000 7fa492600640 1 -- 172.21.15.12:0/3637655290 shutdown_connections 2024-03-17T23:57:42.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.622+0000 7fa492600640 1 --2- 172.21.15.12:0/3637655290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa48c1031e0 0x7fa48c109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.622+0000 7fa492600640 1 --2- 172.21.15.12:0/3637655290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa48c102820 0x7fa48c102ca0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.622+0000 7fa492600640 1 --2- 172.21.15.12:0/3637655290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa48c101630 0x7fa48c101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.622+0000 7fa492600640 1 -- 172.21.15.12:0/3637655290 >> 172.21.15.12:0/3637655290 conn(0x7fa48c0fcbc0 msgr2=0x7fa48c0fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:42.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.622+0000 7fa492600640 1 -- 172.21.15.12:0/3637655290 shutdown_connections 2024-03-17T23:57:42.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.622+0000 7fa492600640 1 -- 172.21.15.12:0/3637655290 wait complete. 2024-03-17T23:57:42.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.623+0000 7fa492600640 1 Processor -- start 2024-03-17T23:57:42.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.623+0000 7fa492600640 1 -- start start 2024-03-17T23:57:42.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa492600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa48c101630 0x7fa48c199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:42.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa492600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa48c102820 0x7fa48c199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:42.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa492600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa48c1031e0 0x7fa48c1a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:42.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa492600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa48c10d2e0 con 0x7fa48c102820 2024-03-17T23:57:42.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa492600640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa48c10d160 con 0x7fa48c1031e0 2024-03-17T23:57:42.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa492600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa48c10d460 con 0x7fa48c101630 2024-03-17T23:57:42.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa491200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa48c101630 0x7fa48c199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:42.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa491200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa48c101630 0x7fa48c199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:39762/0 (socket says 172.21.15.12:39762) 2024-03-17T23:57:42.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa491200640 1 -- 172.21.15.12:0/216546243 learned_addr learned my addr 172.21.15.12:0/216546243 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:42.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa491c00640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa48c1031e0 0x7fa48c1a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:42.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa48be00640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa48c102820 0x7fa48c199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:42.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa491200640 1 -- 172.21.15.12:0/216546243 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa48c1031e0 msgr2=0x7fa48c1a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:42.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa491200640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa48c1031e0 0x7fa48c1a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa491200640 1 -- 172.21.15.12:0/216546243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa48c102820 msgr2=0x7fa48c199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:42.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa491200640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa48c102820 0x7fa48c199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa491200640 1 -- 172.21.15.12:0/216546243 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa48c1a1690 con 0x7fa48c101630 2024-03-17T23:57:42.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa491c00640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa48c1031e0 0x7fa48c1a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:42.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.624+0000 7fa48be00640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa48c102820 0x7fa48c199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:42.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.625+0000 7fa491200640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa48c101630 0x7fa48c199a70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa48000e9c0 tx=0x7fa48000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:42.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.625+0000 7fa489600640 1 -- 172.21.15.12:0/216546243 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48000cd80 con 0x7fa48c101630 2024-03-17T23:57:42.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.625+0000 7fa489600640 1 -- 172.21.15.12:0/216546243 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4800035f0 con 0x7fa48c101630 2024-03-17T23:57:42.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.625+0000 7fa489600640 1 -- 172.21.15.12:0/216546243 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480010690 con 0x7fa48c101630 2024-03-17T23:57:42.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.625+0000 7fa492600640 1 -- 172.21.15.12:0/216546243 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa48c1a1980 con 0x7fa48c101630 2024-03-17T23:57:42.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.625+0000 7fa492600640 1 -- 172.21.15.12:0/216546243 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa48c1a1e50 con 0x7fa48c101630 2024-03-17T23:57:42.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.626+0000 7fa492600640 1 -- 172.21.15.12:0/216546243 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa48c1a20e0 con 0x7fa48c101630 2024-03-17T23:57:42.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.628+0000 7fa489600640 1 -- 172.21.15.12:0/216546243 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa4800037a0 con 0x7fa48c101630 2024-03-17T23:57:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.629+0000 7fa489600640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4740a2880 0x7fa4740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.630+0000 7fa489600640 1 -- 172.21.15.12:0/216546243 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fa4800cf220 con 0x7fa48c101630 2024-03-17T23:57:42.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.630+0000 7fa48be00640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4740a2880 0x7fa4740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:42.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.631+0000 7fa48be00640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4740a2880 0x7fa4740a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa47802c250 tx=0x7fa478005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:42.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.633+0000 7fa489600640 1 -- 172.21.15.12:0/216546243 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa48009bee0 con 0x7fa48c101630 2024-03-17T23:57:42.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.876+0000 7fa492600640 1 -- 172.21.15.12:0/216546243 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa48c1a2960 con 0x7fa48c101630 2024-03-17T23:57:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.877+0000 7fa489600640 1 -- 172.21.15.12:0/216546243 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11936 (secure 0 0 0) 0x7fa48009b7e0 con 0x7fa48c101630 2024-03-17T23:57:42.879 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:42.879 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-17T23:54:56.312554+0000","modified":"2024-03-17T23:55:28.030481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24575,"mds_2":24530,"mds_3":24529,"mds_4":24514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.12:6841/2073607248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":2073607248},{"type":"v1","addr":"172.21.15.12:6841","nonce":2073607248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/2478742711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":2478742711},{"type":"v1","addr":"172.21.15.79:6839","nonce":2478742711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24575,24587,24530,24529,24514]},"id":1}]} 2024-03-17T23:57:42.880 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 28 2024-03-17T23:57:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.880+0000 7fa492600640 1 -- 172.21.15.12:0/216546243 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4740a2880 msgr2=0x7fa4740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.880+0000 7fa492600640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4740a2880 0x7fa4740a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa47802c250 tx=0x7fa478005fd0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.880+0000 7fa492600640 1 -- 172.21.15.12:0/216546243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa48c101630 msgr2=0x7fa48c199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.880+0000 7fa492600640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa48c101630 0x7fa48c199a70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa48000e9c0 tx=0x7fa48000ee80 comp rx=0 tx=0).stop 2024-03-17T23:57:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.880+0000 7fa492600640 1 -- 172.21.15.12:0/216546243 shutdown_connections 2024-03-17T23:57:42.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.880+0000 7fa492600640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa4740a2880 0x7fa4740a4d40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.880+0000 7fa492600640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa48c1031e0 0x7fa48c1a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.880+0000 7fa492600640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa48c102820 0x7fa48c199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.880+0000 7fa492600640 1 --2- 172.21.15.12:0/216546243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa48c101630 0x7fa48c199a70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:42.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.880+0000 7fa492600640 1 -- 172.21.15.12:0/216546243 >> 172.21.15.12:0/216546243 conn(0x7fa48c0fcbc0 msgr2=0x7fa48c073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:42.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.881+0000 7fa492600640 1 -- 172.21.15.12:0/216546243 shutdown_connections 2024-03-17T23:57:42.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:42.881+0000 7fa492600640 1 -- 172.21.15.12:0/216546243 wait complete. 2024-03-17T23:57:42.903 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:57:43.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.102+0000 7f10df200640 1 -- 172.21.15.12:0/1608754161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d81004f0 msgr2=0x7f10d8100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.102+0000 7f10df200640 1 --2- 172.21.15.12:0/1608754161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d81004f0 0x7f10d8100910 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009900 tx=0x7f10cc02bc40 comp rx=0 tx=0).stop 2024-03-17T23:57:43.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.103+0000 7f10df200640 1 -- 172.21.15.12:0/1608754161 shutdown_connections 2024-03-17T23:57:43.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.103+0000 7f10df200640 1 --2- 172.21.15.12:0/1608754161 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10d81020a0 0x7f10d8108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.103+0000 7f10df200640 1 --2- 172.21.15.12:0/1608754161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10d81016e0 0x7f10d8101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.103+0000 7f10df200640 1 --2- 172.21.15.12:0/1608754161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d81004f0 0x7f10d8100910 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.103+0000 7f10df200640 1 -- 172.21.15.12:0/1608754161 >> 172.21.15.12:0/1608754161 conn(0x7f10d80fba60 msgr2=0x7f10d80fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.103+0000 7f10df200640 1 -- 172.21.15.12:0/1608754161 shutdown_connections 2024-03-17T23:57:43.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.103+0000 7f10df200640 1 -- 172.21.15.12:0/1608754161 wait complete. 2024-03-17T23:57:43.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.104+0000 7f10df200640 1 Processor -- start 2024-03-17T23:57:43.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.104+0000 7f10df200640 1 -- start start 2024-03-17T23:57:43.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.104+0000 7f10df200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10d81004f0 0x7f10d8199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.104+0000 7f10df200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d81016e0 0x7f10d819a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.104+0000 7f10df200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10d81020a0 0x7f10d81a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.104+0000 7f10df200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f10d810c1d0 con 0x7f10d81004f0 2024-03-17T23:57:43.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.104+0000 7f10df200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f10d810c050 con 0x7f10d81020a0 2024-03-17T23:57:43.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.104+0000 7f10df200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f10d810c350 con 0x7f10d81016e0 2024-03-17T23:57:43.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.105+0000 7f10dde00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10d81004f0 0x7f10d8199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.105+0000 7f10dd400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d81016e0 0x7f10d819a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.105+0000 7f10dde00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10d81004f0 0x7f10d8199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42024/0 (socket says 172.21.15.12:42024) 2024-03-17T23:57:43.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.105+0000 7f10dd400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d81016e0 0x7f10d819a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:39766/0 (socket says 172.21.15.12:39766) 2024-03-17T23:57:43.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.105+0000 7f10dde00640 1 -- 172.21.15.12:0/1382538633 learned_addr learned my addr 172.21.15.12:0/1382538633 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:43.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.105+0000 7f10de800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10d81020a0 0x7f10d81a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.105+0000 7f10dd400640 1 -- 172.21.15.12:0/1382538633 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10d81020a0 msgr2=0x7f10d81a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.105+0000 7f10dd400640 1 --2- 172.21.15.12:0/1382538633 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10d81020a0 0x7f10d81a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.105+0000 7f10dd400640 1 -- 172.21.15.12:0/1382538633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10d81004f0 msgr2=0x7f10d8199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.105+0000 7f10dd400640 1 --2- 172.21.15.12:0/1382538633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10d81004f0 0x7f10d8199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.105+0000 7f10dd400640 1 -- 172.21.15.12:0/1382538633 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d81a16e0 con 0x7f10d81016e0 2024-03-17T23:57:43.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.105+0000 7f10dde00640 1 --2- 172.21.15.12:0/1382538633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10d81004f0 0x7f10d8199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:43.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.106+0000 7f10dd400640 1 --2- 172.21.15.12:0/1382538633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d81016e0 0x7f10d819a000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f10c800e9f0 tx=0x7f10c800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.106+0000 7f10d2a00640 1 -- 172.21.15.12:0/1382538633 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c800ccc0 con 0x7f10d81016e0 2024-03-17T23:57:43.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.106+0000 7f10d2a00640 1 -- 172.21.15.12:0/1382538633 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10c8010910 con 0x7f10d81016e0 2024-03-17T23:57:43.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.106+0000 7f10d2a00640 1 -- 172.21.15.12:0/1382538633 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c8018960 con 0x7f10d81016e0 2024-03-17T23:57:43.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.106+0000 7f10df200640 1 -- 172.21.15.12:0/1382538633 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10d81a19d0 con 0x7f10d81016e0 2024-03-17T23:57:43.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.106+0000 7f10df200640 1 -- 172.21.15.12:0/1382538633 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10d81a1f70 con 0x7f10d81016e0 2024-03-17T23:57:43.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.107+0000 7f10df200640 1 -- 172.21.15.12:0/1382538633 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10d81a21a0 con 0x7f10d81016e0 2024-03-17T23:57:43.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.109+0000 7f10d2a00640 1 -- 172.21.15.12:0/1382538633 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f10c8010ac0 con 0x7f10d81016e0 2024-03-17T23:57:43.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.110+0000 7f10d2a00640 1 --2- 172.21.15.12:0/1382538633 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f10c00a2880 0x7f10c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.111+0000 7f10d2a00640 1 -- 172.21.15.12:0/1382538633 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f10c80cf0f0 con 0x7f10d81016e0 2024-03-17T23:57:43.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.111+0000 7f10dde00640 1 --2- 172.21.15.12:0/1382538633 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f10c00a2880 0x7f10c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.111+0000 7f10dde00640 1 --2- 172.21.15.12:0/1382538633 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f10c00a2880 0x7f10c00a4d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f10cc005ab0 tx=0x7f10cc02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.114+0000 7f10d2a00640 1 -- 172.21.15.12:0/1382538633 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f10c809b7c0 con 0x7f10d81016e0 2024-03-17T23:57:43.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:42 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/570915636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:57:43.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:42 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2580155406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:57:43.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:42 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/216546243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:43.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:42 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/570915636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:57:43.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:42 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2580155406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:57:43.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:42 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/216546243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:43.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.320+0000 7f10df200640 1 -- 172.21.15.12:0/1382538633 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10d81a2530 con 0x7f10d81016e0 2024-03-17T23:57:43.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.323+0000 7f10d2a00640 1 -- 172.21.15.12:0/1382538633 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24618 (secure 0 0 0) 0x7f10d81a2530 con 0x7f10d81016e0 2024-03-17T23:57:43.325 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-17T23:57:43.325 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":94,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:56:55.719277+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:57:43.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.326+0000 7f10df200640 1 -- 172.21.15.12:0/1382538633 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f10c00a2880 msgr2=0x7f10c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.326+0000 7f10df200640 1 --2- 172.21.15.12:0/1382538633 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f10c00a2880 0x7f10c00a4d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f10cc005ab0 tx=0x7f10cc02cf90 comp rx=0 tx=0).stop 2024-03-17T23:57:43.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.326+0000 7f10df200640 1 -- 172.21.15.12:0/1382538633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d81016e0 msgr2=0x7f10d819a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.326+0000 7f10df200640 1 --2- 172.21.15.12:0/1382538633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d81016e0 0x7f10d819a000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f10c800e9f0 tx=0x7f10c800eeb0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.326+0000 7f10df200640 1 -- 172.21.15.12:0/1382538633 shutdown_connections 2024-03-17T23:57:43.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.326+0000 7f10df200640 1 --2- 172.21.15.12:0/1382538633 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f10c00a2880 0x7f10c00a4d40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.326+0000 7f10df200640 1 --2- 172.21.15.12:0/1382538633 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10d81020a0 0x7f10d81a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.326+0000 7f10df200640 1 --2- 172.21.15.12:0/1382538633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d81016e0 0x7f10d819a000 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.326+0000 7f10df200640 1 --2- 172.21.15.12:0/1382538633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10d81004f0 0x7f10d8199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.326+0000 7f10df200640 1 -- 172.21.15.12:0/1382538633 >> 172.21.15.12:0/1382538633 conn(0x7f10d80fba60 msgr2=0x7f10d80fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.326+0000 7f10df200640 1 -- 172.21.15.12:0/1382538633 shutdown_connections 2024-03-17T23:57:43.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.326+0000 7f10df200640 1 -- 172.21.15.12:0/1382538633 wait complete. 2024-03-17T23:57:43.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:42 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/570915636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-17T23:57:43.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:42 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2580155406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-17T23:57:43.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:42 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/216546243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-17T23:57:43.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-17T23:54:56.312554+0000', 'modified': '2024-03-17T23:55:28.030481+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24587, 'mds_1': 24575, 'mds_2': 24530, 'mds_3': 24529, 'mds_4': 24514}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24439': {'gid': 24439, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6833/1787103083', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6832', 'nonce': 1787103083}, {'type': 'v1', 'addr': '172.21.15.191:6833', 'nonce': 1787103083}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.79:6833/843208408', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6832', 'nonce': 843208408}, {'type': 'v1', 'addr': '172.21.15.79:6833', 'nonce': 843208408}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'd', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.12:6837/1104379442', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6836', 'nonce': 1104379442}, {'type': 'v1', 'addr': '172.21.15.12:6837', 'nonce': 1104379442}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24502': {'gid': 24502, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6835/3192827956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 3192827956}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 3192827956}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.79:6835/4099928196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6834', 'nonce': 4099928196}, {'type': 'v1', 'addr': '172.21.15.79:6835', 'nonce': 4099928196}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24514': {'gid': 24514, 'name': 'c', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.191:6837/2977770216', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 2977770216}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 2977770216}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24529': {'gid': 24529, 'name': 'i', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.191:6839/4148729123', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6838', 'nonce': 4148729123}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 4148729123}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.79:6837/3814468390', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6836', 'nonce': 3814468390}, {'type': 'v1', 'addr': '172.21.15.79:6837', 'nonce': 3814468390}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24575': {'gid': 24575, 'name': 'a', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.12:6841/2073607248', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.12:6840', 'nonce': 2073607248}, {'type': 'v1', 'addr': '172.21.15.12:6841', 'nonce': 2073607248}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.79:6839/2478742711', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6838', 'nonce': 2478742711}, {'type': 'v1', 'addr': '172.21.15.79:6839', 'nonce': 2478742711}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24587, 'qdb_cluster': [24575, 24587, 24530, 24529, 24514]} 2024-03-17T23:57:43.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-17T23:57:43.348 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-17T23:57:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.563+0000 7f6eaf000640 1 -- 172.21.15.12:0/811652510 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ea8104a70 msgr2=0x7f6ea8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.563+0000 7f6eaf000640 1 --2- 172.21.15.12:0/811652510 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ea8104a70 0x7f6ea8106e60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6e94009900 tx=0x7f6e9402bc30 comp rx=0 tx=0).stop 2024-03-17T23:57:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.563+0000 7f6eaf000640 1 -- 172.21.15.12:0/811652510 shutdown_connections 2024-03-17T23:57:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.564+0000 7f6eaf000640 1 --2- 172.21.15.12:0/811652510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ea81073a0 0x7f6ea81097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.564+0000 7f6eaf000640 1 --2- 172.21.15.12:0/811652510 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ea8104a70 0x7f6ea8106e60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.564+0000 7f6eaf000640 1 --2- 172.21.15.12:0/811652510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ea8102140 0x7f6ea8104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.564+0000 7f6eaf000640 1 -- 172.21.15.12:0/811652510 >> 172.21.15.12:0/811652510 conn(0x7f6ea80fbb50 msgr2=0x7f6ea80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.564+0000 7f6eaf000640 1 -- 172.21.15.12:0/811652510 shutdown_connections 2024-03-17T23:57:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.564+0000 7f6eaf000640 1 -- 172.21.15.12:0/811652510 wait complete. 2024-03-17T23:57:43.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.565+0000 7f6eaf000640 1 Processor -- start 2024-03-17T23:57:43.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.565+0000 7f6eaf000640 1 -- start start 2024-03-17T23:57:43.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.565+0000 7f6eaf000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ea8102140 0x7f6ea8199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.565+0000 7f6eaf000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ea8104a70 0x7f6ea819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.565+0000 7f6eaf000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ea81073a0 0x7f6ea81a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.565+0000 7f6eaf000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea810d390 con 0x7f6ea8102140 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.565+0000 7f6eaf000640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea810d210 con 0x7f6ea81073a0 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.565+0000 7f6eaf000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea810d510 con 0x7f6ea8104a70 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.565+0000 7f6ead200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ea8104a70 0x7f6ea819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.565+0000 7f6eadc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ea8102140 0x7f6ea8199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.565+0000 7f6eae600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ea81073a0 0x7f6ea81a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.565+0000 7f6eadc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ea8102140 0x7f6ea8199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42040/0 (socket says 172.21.15.12:42040) 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.565+0000 7f6eadc00640 1 -- 172.21.15.12:0/3378747930 learned_addr learned my addr 172.21.15.12:0/3378747930 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.566+0000 7f6eadc00640 1 -- 172.21.15.12:0/3378747930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ea8104a70 msgr2=0x7f6ea819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.566+0000 7f6eadc00640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ea8104a70 0x7f6ea819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.566+0000 7f6eadc00640 1 -- 172.21.15.12:0/3378747930 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ea81073a0 msgr2=0x7f6ea81a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.566+0000 7f6eadc00640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ea81073a0 0x7f6ea81a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.566+0000 7f6eadc00640 1 -- 172.21.15.12:0/3378747930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea81a1830 con 0x7f6ea8102140 2024-03-17T23:57:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.566+0000 7f6ead200640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ea8104a70 0x7f6ea819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.566+0000 7f6eae600640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ea81073a0 0x7f6ea81a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.566+0000 7f6eadc00640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ea8102140 0x7f6ea8199c20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00d950 tx=0x7f6e9c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.566+0000 7f6ea6000640 1 -- 172.21.15.12:0/3378747930 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c017070 con 0x7f6ea8102140 2024-03-17T23:57:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.567+0000 7f6eaf000640 1 -- 172.21.15.12:0/3378747930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea81a1b20 con 0x7f6ea8102140 2024-03-17T23:57:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.567+0000 7f6ea6000640 1 -- 172.21.15.12:0/3378747930 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c009c30 con 0x7f6ea8102140 2024-03-17T23:57:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.567+0000 7f6ea6000640 1 -- 172.21.15.12:0/3378747930 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c020060 con 0x7f6ea8102140 2024-03-17T23:57:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.567+0000 7f6eaf000640 1 -- 172.21.15.12:0/3378747930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea81a2070 con 0x7f6ea8102140 2024-03-17T23:57:43.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.570+0000 7f6eb3377740 1 Processor -- start 2024-03-17T23:57:43.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.570+0000 7f6ea6000640 1 -- 172.21.15.12:0/3378747930 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6e9c0109d0 con 0x7f6ea8102140 2024-03-17T23:57:43.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.570+0000 7f6eb3377740 1 -- start start 2024-03-17T23:57:43.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.570+0000 7f6eb3377740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6f340 0x557cf6c6f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.570+0000 7f6eb3377740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c6fd30 0x557cf6c74830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6eb3377740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c74ea0 0x557cf6c77330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6eb3377740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557cf6bd3c50 con 0x557cf6c6f340 2024-03-17T23:57:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6eb3377740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x557cf6b7c9d0 con 0x557cf6c74ea0 2024-03-17T23:57:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6eb3377740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557cf6bcfcf0 con 0x557cf6c6fd30 2024-03-17T23:57:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6eae600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6f340 0x557cf6c6f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6eadc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c6fd30 0x557cf6c74830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6ead200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c74ea0 0x557cf6c77330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6ead200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c74ea0 0x557cf6c77330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36508/0 (socket says 172.21.15.12:36508) 2024-03-17T23:57:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6ead200640 1 -- 172.21.15.12:0/3636180393 learned_addr learned my addr 172.21.15.12:0/3636180393 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6eae600640 1 --2- 172.21.15.12:0/3636180393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6f340 0x557cf6c6f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42052/0 (socket says 172.21.15.12:42052) 2024-03-17T23:57:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6eadc00640 1 -- 172.21.15.12:0/3636180393 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c74ea0 msgr2=0x557cf6c77330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6eadc00640 1 --2- 172.21.15.12:0/3636180393 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c74ea0 0x557cf6c77330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6eadc00640 1 -- 172.21.15.12:0/3636180393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6f340 msgr2=0x557cf6c6f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6eadc00640 1 --2- 172.21.15.12:0/3636180393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6f340 0x557cf6c6f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6eadc00640 1 -- 172.21.15.12:0/3636180393 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cf6b96f10 con 0x557cf6c6fd30 2024-03-17T23:57:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.571+0000 7f6ead200640 1 --2- 172.21.15.12:0/3636180393 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c74ea0 0x557cf6c77330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.572+0000 7f6eadc00640 1 --2- 172.21.15.12:0/3636180393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c6fd30 0x557cf6c74830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0cc1f0 tx=0x7f6e9c0d7580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91b29d2f0412dd94 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.572+0000 7f6e9aa00640 1 -- 172.21.15.12:0/3636180393 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0dc630 con 0x557cf6c6fd30 2024-03-17T23:57:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.572+0000 7f6e9aa00640 1 -- 172.21.15.12:0/3636180393 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c0d2e10 con 0x557cf6c6fd30 2024-03-17T23:57:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.572+0000 7f6e9aa00640 1 -- 172.21.15.12:0/3636180393 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0dba00 con 0x557cf6c6fd30 2024-03-17T23:57:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.572+0000 7f6ea6000640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6e900a2800 0x7f6e900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.573+0000 7f6ea6000640 1 -- 172.21.15.12:0/3378747930 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6e9c0ceb30 con 0x7f6ea8102140 2024-03-17T23:57:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.573+0000 7f6eb3377740 1 -- 172.21.15.12:0/3636180393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c6fd30 msgr2=0x557cf6c74830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.573+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3636180393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c6fd30 0x557cf6c74830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0cc1f0 tx=0x7f6e9c0d7580 comp rx=0 tx=0).stop 2024-03-17T23:57:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.573+0000 7f6eae600640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6e900a2800 0x7f6e900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.573+0000 7f6eb3377740 1 -- 172.21.15.12:0/3636180393 shutdown_connections 2024-03-17T23:57:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.573+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3636180393 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c74ea0 0x557cf6c77330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.573+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3636180393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c6fd30 0x557cf6c74830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.573+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3636180393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6f340 0x557cf6c6f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.573+0000 7f6eb3377740 1 -- 172.21.15.12:0/3636180393 >> 172.21.15.12:0/3636180393 conn(0x557cf6c6a8b0 msgr2=0x557cf6c6acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.573+0000 7f6eb3377740 1 -- 172.21.15.12:0/3636180393 shutdown_connections 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.573+0000 7f6eb3377740 1 -- 172.21.15.12:0/3636180393 wait complete. 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.574+0000 7f6eae600640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6e900a2800 0x7f6e900a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x557cf6c70590 tx=0x7f6ea0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.574+0000 7f6eb3377740 1 -- start start 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.574+0000 7f6eb3377740 1 Processor -- start 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.574+0000 7f6eb3377740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 0x557cf6c794c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.574+0000 7f6eb3377740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6fd30 0x557cf6c79a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.574+0000 7f6eb3377740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 0x557cf6c80a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.574+0000 7f6eb3377740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557cf6c77e50 con 0x557cf6c6fd30 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.574+0000 7f6eb3377740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x557cf6b8fe20 con 0x557cf6c6f340 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.574+0000 7f6eb3377740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557cf6c77fd0 con 0x557cf6c74ea0 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.574+0000 7f6eb3377740 10 client.0 _ll_register_callbacks cb 0x557cf6b8f7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6eadc00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 0x557cf6c794c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6eadc00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 0x557cf6c794c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36522/0 (socket says 172.21.15.12:36522) 2024-03-17T23:57:43.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6eadc00640 1 -- 172.21.15.12:0/740374360 learned_addr learned my addr 172.21.15.12:0/740374360 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:43.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6eae600640 1 --2- 172.21.15.12:0/740374360 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 0x557cf6c80a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6ead200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6fd30 0x557cf6c79a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6eadc00640 1 -- 172.21.15.12:0/740374360 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 msgr2=0x557cf6c80a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6eadc00640 1 --2- 172.21.15.12:0/740374360 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 0x557cf6c80a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6eadc00640 1 -- 172.21.15.12:0/740374360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6fd30 msgr2=0x557cf6c79a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6eadc00640 1 --2- 172.21.15.12:0/740374360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6fd30 0x557cf6c79a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6eadc00640 1 -- 172.21.15.12:0/740374360 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cf68e82d0 con 0x557cf6c6f340 2024-03-17T23:57:43.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6eadc00640 1 --2- 172.21.15.12:0/740374360 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 0x557cf6c794c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c097ae0 tx=0x7f6e9c005bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed3f5263fb79fd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6e8f400640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-03-17T23:57:43.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6e8f400640 1 -- 172.21.15.12:0/740374360 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0973b0 con 0x557cf6c6f340 2024-03-17T23:57:43.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.575+0000 7f6e8f400640 1 -- 172.21.15.12:0/740374360 <== mon.2 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6e9c09b400 con 0x557cf6c6f340 2024-03-17T23:57:43.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.576+0000 7f6eb3377740 1 -- 172.21.15.12:0/740374360 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e9c0d2e10 con 0x557cf6c6f340 2024-03-17T23:57:43.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.576+0000 7f6e8f400640 1 -- 172.21.15.12:0/740374360 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6e9c0992e0 con 0x557cf6c6f340 2024-03-17T23:57:43.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.576+0000 7f6eb3377740 10 client.25370 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:43.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.576+0000 7f6eb3377740 1 -- 172.21.15.12:0/740374360 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557cf69b26b0 con 0x557cf6c6f340 2024-03-17T23:57:43.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.576+0000 7f6e8f400640 1 -- 172.21.15.12:0/740374360 <== mon.2 v2:172.21.15.79:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c0d2e10 con 0x557cf6c6f340 2024-03-17T23:57:43.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.576+0000 7f6e8f400640 1 -- 172.21.15.12:0/740374360 <== mon.2 v2:172.21.15.79:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6e9c023020 con 0x557cf6c6f340 2024-03-17T23:57:43.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.576+0000 7f6eb3377740 10 client.25370 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:43.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.576+0000 7f6eb3377740 10 client.25370 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-17T23:57:43.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.576+0000 7f6eb3377740 20 client.25370 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:43.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.576+0000 7f6eb3377740 1 --2- 172.21.15.12:0/740374360 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x557cf6c84f30 0x557cf6c87030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.576+0000 7f6eb3377740 4 client.25370 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:57:43.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.576+0000 7f6eb3377740 1 -- 172.21.15.12:0/740374360 --> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557cf6c312b0 con 0x557cf6c84f30 2024-03-17T23:57:43.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.577+0000 7f6ead200640 1 --2- 172.21.15.12:0/740374360 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x557cf6c84f30 0x557cf6c87030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.577+0000 7f6ead200640 1 --2- 172.21.15.12:0/740374360 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x557cf6c84f30 0x557cf6c87030 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1c5d1d8759073e6a server_cookie=7094fc7c48466d37 in_seq=0 out_seq=0 2024-03-17T23:57:43.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.577+0000 7f6e8f400640 10 client.25370 ms_handle_connect on v2:172.21.15.191:6836/2977770216 2024-03-17T23:57:43.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.578+0000 7f6e8f400640 1 -- 172.21.15.12:0/740374360 <== mds.4 v2:172.21.15.191:6836/2977770216 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557cf6c312b0 con 0x557cf6c84f30 2024-03-17T23:57:43.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.578+0000 7f6e8f400640 10 client.25370 handle_command_reply: tid=0 2024-03-17T23:57:43.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.578+0000 7f6eb3377740 1 client.25370 shutdown 2024-03-17T23:57:43.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.578+0000 7f6eb3377740 1 -- 172.21.15.12:0/740374360 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 msgr2=0x557cf6c794c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.578+0000 7f6eb3377740 1 --2- 172.21.15.12:0/740374360 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 0x557cf6c794c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c097ae0 tx=0x7f6e9c005bb0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.578+0000 7f6eb3377740 1 -- 172.21.15.12:0/740374360 shutdown_connections 2024-03-17T23:57:43.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.578+0000 7f6eb3377740 1 --2- 172.21.15.12:0/740374360 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x557cf6c84f30 0x557cf6c87030 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.578+0000 7f6eb3377740 1 --2- 172.21.15.12:0/740374360 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 0x557cf6c80a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.578+0000 7f6eb3377740 1 --2- 172.21.15.12:0/740374360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6fd30 0x557cf6c79a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.578+0000 7f6e8f400640 0 client.25370 ms_handle_reset on v2:172.21.15.191:6836/2977770216 2024-03-17T23:57:43.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.578+0000 7f6eb3377740 1 --2- 172.21.15.12:0/740374360 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 0x557cf6c794c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.578+0000 7f6eb3377740 1 -- 172.21.15.12:0/740374360 >> 172.21.15.12:0/740374360 conn(0x557cf6c6a8b0 msgr2=0x557cf6c69a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.578+0000 7f6eb3377740 1 -- 172.21.15.12:0/740374360 shutdown_connections 2024-03-17T23:57:43.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.579+0000 7f6eb3377740 1 -- 172.21.15.12:0/740374360 wait complete. 2024-03-17T23:57:43.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.579+0000 7f6eb3377740 20 client.25370 trim_cache size 0 max 16384 2024-03-17T23:57:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.598+0000 7f6eb3377740 1 Processor -- start 2024-03-17T23:57:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.598+0000 7f6eb3377740 1 -- start start 2024-03-17T23:57:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6eb3377740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6f340 0x557cf6c797e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6eb3377740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6fd30 0x557cf6c80a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6eb3377740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 0x557cf6c691e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6eb3377740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557cf6c85370 con 0x557cf6c6f340 2024-03-17T23:57:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6eb3377740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x557cf6c85a40 con 0x557cf6c6fd30 2024-03-17T23:57:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6eb3377740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557cf6c86110 con 0x557cf6c74ea0 2024-03-17T23:57:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6ead200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6fd30 0x557cf6c80a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6eadc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6f340 0x557cf6c797e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6eae600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 0x557cf6c691e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6eadc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6f340 0x557cf6c797e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42068/0 (socket says 172.21.15.12:42068) 2024-03-17T23:57:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6ead200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6fd30 0x557cf6c80a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36538/0 (socket says 172.21.15.12:36538) 2024-03-17T23:57:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6eadc00640 1 -- 172.21.15.12:0/3929253268 learned_addr learned my addr 172.21.15.12:0/3929253268 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6ead200640 1 -- 172.21.15.12:0/3929253268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 msgr2=0x557cf6c691e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6ead200640 1 --2- 172.21.15.12:0/3929253268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 0x557cf6c691e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6ead200640 1 -- 172.21.15.12:0/3929253268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6f340 msgr2=0x557cf6c797e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6ead200640 1 --2- 172.21.15.12:0/3929253268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6f340 0x557cf6c797e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.599+0000 7f6ead200640 1 -- 172.21.15.12:0/3929253268 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cf6c70110 con 0x557cf6c6fd30 2024-03-17T23:57:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.600+0000 7f6ead200640 1 --2- 172.21.15.12:0/3929253268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6fd30 0x557cf6c80a00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6e94000c00 tx=0x7f6e9402df90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=985665903ef4b513 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.600+0000 7f6e9a000640 1 -- 172.21.15.12:0/3929253268 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9404f070 con 0x557cf6c6fd30 2024-03-17T23:57:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.600+0000 7f6e9a000640 1 -- 172.21.15.12:0/3929253268 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9404a410 con 0x557cf6c6fd30 2024-03-17T23:57:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.600+0000 7f6e9a000640 1 -- 172.21.15.12:0/3929253268 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e940046c0 con 0x557cf6c6fd30 2024-03-17T23:57:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.601+0000 7f6eb3377740 1 -- 172.21.15.12:0/3929253268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6fd30 msgr2=0x557cf6c80a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.601+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3929253268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6fd30 0x557cf6c80a00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6e94000c00 tx=0x7f6e9402df90 comp rx=0 tx=0).stop 2024-03-17T23:57:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.601+0000 7f6eb3377740 1 -- 172.21.15.12:0/3929253268 shutdown_connections 2024-03-17T23:57:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.601+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3929253268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 0x557cf6c691e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.601+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3929253268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6fd30 0x557cf6c80a00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.601+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3929253268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6f340 0x557cf6c797e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.601+0000 7f6eb3377740 1 -- 172.21.15.12:0/3929253268 >> 172.21.15.12:0/3929253268 conn(0x557cf6c6a8b0 msgr2=0x557cf6c790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.601+0000 7f6eb3377740 1 -- 172.21.15.12:0/3929253268 shutdown_connections 2024-03-17T23:57:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.601+0000 7f6eb3377740 1 -- 172.21.15.12:0/3929253268 wait complete. 2024-03-17T23:57:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.601+0000 7f6eb3377740 1 -- start start 2024-03-17T23:57:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6eb3377740 1 Processor -- start 2024-03-17T23:57:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6eb3377740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 0x557cf6c7c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6eb3377740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6fd30 0x557cf6c7cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6eb3377740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 0x557cf6c7fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6eb3377740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557cf6c86eb0 con 0x557cf6c6fd30 2024-03-17T23:57:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6eb3377740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x557cf6c867e0 con 0x557cf6c6f340 2024-03-17T23:57:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6eb3377740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557cf6c87580 con 0x557cf6c74ea0 2024-03-17T23:57:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6eb3377740 10 client.0 _ll_register_callbacks cb 0x7f6e9c0d2ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6ead200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6fd30 0x557cf6c7cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6eadc00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 0x557cf6c7c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6eae600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 0x557cf6c7fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6eadc00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 0x557cf6c7c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36544/0 (socket says 172.21.15.12:36544) 2024-03-17T23:57:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6eae600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 0x557cf6c7fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:39814/0 (socket says 172.21.15.12:39814) 2024-03-17T23:57:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6ead200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6fd30 0x557cf6c7cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42080/0 (socket says 172.21.15.12:42080) 2024-03-17T23:57:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.602+0000 7f6eadc00640 1 -- 172.21.15.12:0/3918423567 learned_addr learned my addr 172.21.15.12:0/3918423567 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.603+0000 7f6ead200640 1 -- 172.21.15.12:0/3918423567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 msgr2=0x557cf6c7fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.603+0000 7f6ead200640 1 --2- 172.21.15.12:0/3918423567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 0x557cf6c7fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.603+0000 7f6ead200640 1 -- 172.21.15.12:0/3918423567 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 msgr2=0x557cf6c7c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.603+0000 7f6ead200640 1 --2- 172.21.15.12:0/3918423567 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 0x557cf6c7c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.603+0000 7f6ead200640 1 -- 172.21.15.12:0/3918423567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cf6c80200 con 0x557cf6c6fd30 2024-03-17T23:57:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.603+0000 7f6eadc00640 1 --2- 172.21.15.12:0/3918423567 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 0x557cf6c7c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.603+0000 7f6ead200640 1 --2- 172.21.15.12:0/3918423567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6fd30 0x557cf6c7cbf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6e9402c170 tx=0x7f6e94004100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec199f86dc0df657 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.603+0000 7f6e8f400640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-03-17T23:57:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.603+0000 7f6e8f400640 1 -- 172.21.15.12:0/3918423567 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e94058070 con 0x557cf6c6fd30 2024-03-17T23:57:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.604+0000 7f6e8f400640 1 -- 172.21.15.12:0/3918423567 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6e9404ad10 con 0x557cf6c6fd30 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.604+0000 7f6eb3377740 1 -- 172.21.15.12:0/3918423567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e9404a410 con 0x557cf6c6fd30 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.604+0000 7f6e8f400640 1 -- 172.21.15.12:0/3918423567 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6e94064050 con 0x557cf6c6fd30 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.604+0000 7f6eb3377740 10 client.15234 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.604+0000 7f6eb3377740 1 -- 172.21.15.12:0/3918423567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557cf6cebea0 con 0x557cf6c6fd30 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.604+0000 7f6e8f400640 1 -- 172.21.15.12:0/3918423567 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e9404a410 con 0x557cf6c6fd30 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.604+0000 7f6e8f400640 1 -- 172.21.15.12:0/3918423567 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6e940051a0 con 0x557cf6c6fd30 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.604+0000 7f6eb3377740 10 client.15234 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.604+0000 7f6eb3377740 10 client.15234 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.604+0000 7f6eb3377740 20 client.15234 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.605+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3918423567 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x557cf6cf0550 0x557cf6ced060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.605+0000 7f6eb3377740 4 client.15234 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.605+0000 7f6eb3377740 1 -- 172.21.15.12:0/3918423567 --> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] -- command(tid 0: {"prefix": "status"}) -- 0x557cf6be1190 con 0x557cf6cf0550 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.605+0000 7f6eadc00640 1 --2- 172.21.15.12:0/3918423567 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x557cf6cf0550 0x557cf6ced060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.605+0000 7f6eadc00640 1 --2- 172.21.15.12:0/3918423567 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x557cf6cf0550 0x557cf6ced060 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1ce72ccf26c181c8 server_cookie=7bc9a60b7843a90b in_seq=0 out_seq=0 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.605+0000 7f6e8f400640 10 client.15234 ms_handle_connect on v2:172.21.15.191:6836/2977770216 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.605+0000 7f6e8f400640 1 -- 172.21.15.12:0/3918423567 <== mds.4 v2:172.21.15.191:6836/2977770216 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557cf6be1190 con 0x557cf6cf0550 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.605+0000 7f6e8f400640 10 client.15234 handle_command_reply: tid=0 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.606+0000 7f6eb3377740 1 client.15234 shutdown 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.606+0000 7f6eb3377740 1 -- 172.21.15.12:0/3918423567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6fd30 msgr2=0x557cf6c7cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.606+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3918423567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6fd30 0x557cf6c7cbf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6e9402c170 tx=0x7f6e94004100 comp rx=0 tx=0).stop 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.606+0000 7f6eb3377740 1 -- 172.21.15.12:0/3918423567 shutdown_connections 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.606+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3918423567 >> [v2:172.21.15.191:6836/2977770216,v1:172.21.15.191:6837/2977770216] conn(0x557cf6cf0550 0x557cf6ced060 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.606+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3918423567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557cf6c74ea0 0x557cf6c7fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.606+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3918423567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557cf6c6fd30 0x557cf6c7cbf0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.606+0000 7f6eb3377740 1 --2- 172.21.15.12:0/3918423567 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557cf6c6f340 0x557cf6c7c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.606+0000 7f6e8f400640 0 client.15234 ms_handle_reset on v2:172.21.15.191:6836/2977770216 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.606+0000 7f6eb3377740 1 -- 172.21.15.12:0/3918423567 >> 172.21.15.12:0/3918423567 conn(0x557cf6c6a8b0 msgr2=0x557cf6c691e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.606+0000 7f6eb3377740 1 -- 172.21.15.12:0/3918423567 shutdown_connections 2024-03-17T23:57:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.606+0000 7f6eb3377740 1 -- 172.21.15.12:0/3918423567 wait complete. 2024-03-17T23:57:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.606+0000 7f6eb3377740 20 client.15234 trim_cache size 0 max 16384 2024-03-17T23:57:43.612 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-17T23:57:43.612 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:57:43.612 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 4, 2024-03-17T23:57:43.612 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24514, 2024-03-17T23:57:43.612 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-17T23:57:43.612 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-17T23:57:43.612 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-17T23:57:43.612 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 136.58425032900001, 2024-03-17T23:57:43.613 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 28, 2024-03-17T23:57:43.613 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 94, 2024-03-17T23:57:43.613 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 90, 2024-03-17T23:57:43.613 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 190.36721760699999 2024-03-17T23:57:43.613 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-17T23:57:43.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.607+0000 7f6eaf000640 1 -- 172.21.15.12:0/3378747930 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6e900a2800 msgr2=0x7f6e900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.607+0000 7f6eaf000640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6e900a2800 0x7f6e900a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x557cf6c70590 tx=0x7f6ea0009500 comp rx=0 tx=0).stop 2024-03-17T23:57:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.607+0000 7f6eaf000640 1 -- 172.21.15.12:0/3378747930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ea8102140 msgr2=0x7f6ea8199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.607+0000 7f6eaf000640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ea8102140 0x7f6ea8199c20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00d950 tx=0x7f6e9c00de10 comp rx=0 tx=0).stop 2024-03-17T23:57:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.607+0000 7f6eaf000640 1 -- 172.21.15.12:0/3378747930 shutdown_connections 2024-03-17T23:57:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.607+0000 7f6eaf000640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f6e900a2800 0x7f6e900a4cc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.607+0000 7f6eaf000640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ea81073a0 0x7f6ea81a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.607+0000 7f6eaf000640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ea8104a70 0x7f6ea819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.607+0000 7f6eaf000640 1 --2- 172.21.15.12:0/3378747930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ea8102140 0x7f6ea8199c20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.607+0000 7f6eaf000640 1 -- 172.21.15.12:0/3378747930 >> 172.21.15.12:0/3378747930 conn(0x7f6ea80fbb50 msgr2=0x7f6ea81018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.607+0000 7f6eaf000640 1 -- 172.21.15.12:0/3378747930 shutdown_connections 2024-03-17T23:57:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.607+0000 7f6eaf000640 1 -- 172.21.15.12:0/3378747930 wait complete. 2024-03-17T23:57:43.625 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-17T23:57:43.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.835+0000 7fb143600640 1 -- 172.21.15.12:0/1790235516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb13c1033a0 msgr2=0x7fb13c109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.835+0000 7fb143600640 1 --2- 172.21.15.12:0/1790235516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb13c1033a0 0x7fb13c109c30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb134009900 tx=0x7fb13402bc30 comp rx=0 tx=0).stop 2024-03-17T23:57:43.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.835+0000 7fb143600640 1 -- 172.21.15.12:0/1790235516 shutdown_connections 2024-03-17T23:57:43.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.835+0000 7fb143600640 1 --2- 172.21.15.12:0/1790235516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb13c1033a0 0x7fb13c109c30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.835+0000 7fb143600640 1 --2- 172.21.15.12:0/1790235516 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb13c1029e0 0x7fb13c102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.835+0000 7fb143600640 1 --2- 172.21.15.12:0/1790235516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13c1017f0 0x7fb13c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.835+0000 7fb143600640 1 -- 172.21.15.12:0/1790235516 >> 172.21.15.12:0/1790235516 conn(0x7fb13c0fcd80 msgr2=0x7fb13c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.836+0000 7fb143600640 1 -- 172.21.15.12:0/1790235516 shutdown_connections 2024-03-17T23:57:43.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.836+0000 7fb143600640 1 -- 172.21.15.12:0/1790235516 wait complete. 2024-03-17T23:57:43.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.837+0000 7fb143600640 1 Processor -- start 2024-03-17T23:57:43.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.837+0000 7fb143600640 1 -- start start 2024-03-17T23:57:43.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.837+0000 7fb143600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb13c1017f0 0x7fb13c199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.837+0000 7fb143600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13c1029e0 0x7fb13c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.837+0000 7fb143600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb13c1033a0 0x7fb13c1a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.837+0000 7fb143600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c10d4c0 con 0x7fb13c1029e0 2024-03-17T23:57:43.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.837+0000 7fb143600640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c10d340 con 0x7fb13c1033a0 2024-03-17T23:57:43.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.837+0000 7fb143600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c10d640 con 0x7fb13c1017f0 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.837+0000 7fb140c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb13c1017f0 0x7fb13c199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.837+0000 7fb140c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb13c1017f0 0x7fb13c199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:40782/0 (socket says 172.21.15.12:40782) 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.837+0000 7fb140c00640 1 -- 172.21.15.12:0/727008331 learned_addr learned my addr 172.21.15.12:0/727008331 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.838+0000 7fb141600640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb13c1033a0 0x7fb13c1a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.838+0000 7fb133e00640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13c1029e0 0x7fb13c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.838+0000 7fb140c00640 1 -- 172.21.15.12:0/727008331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb13c1033a0 msgr2=0x7fb13c1a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.838+0000 7fb140c00640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb13c1033a0 0x7fb13c1a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.838+0000 7fb140c00640 1 -- 172.21.15.12:0/727008331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13c1029e0 msgr2=0x7fb13c19a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.838+0000 7fb140c00640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13c1029e0 0x7fb13c19a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.838+0000 7fb140c00640 1 -- 172.21.15.12:0/727008331 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c1a18d0 con 0x7fb13c1017f0 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.838+0000 7fb133e00640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13c1029e0 0x7fb13c19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.838+0000 7fb140c00640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb13c1017f0 0x7fb13c199c90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009fd0 tx=0x7fb12c00c270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.839+0000 7fb13a000640 1 -- 172.21.15.12:0/727008331 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c011470 con 0x7fb13c1017f0 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.839+0000 7fb13a000640 1 -- 172.21.15.12:0/727008331 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb12c011af0 con 0x7fb13c1017f0 2024-03-17T23:57:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.839+0000 7fb13a000640 1 -- 172.21.15.12:0/727008331 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c01ab90 con 0x7fb13c1017f0 2024-03-17T23:57:43.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.839+0000 7fb143600640 1 -- 172.21.15.12:0/727008331 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13c1a1bc0 con 0x7fb13c1017f0 2024-03-17T23:57:43.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.839+0000 7fb143600640 1 -- 172.21.15.12:0/727008331 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb13c1a2160 con 0x7fb13c1017f0 2024-03-17T23:57:43.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.842+0000 7fb1463d9740 1 Processor -- start 2024-03-17T23:57:43.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.842+0000 7fb1463d9740 1 -- start start 2024-03-17T23:57:43.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.842+0000 7fb1463d9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb87160 0x55c10cb87580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb1463d9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87b50 0x55c10cb8c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb1463d9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb8ccc0 0x55c10cb8f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb1463d9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c10caebc50 con 0x55c10cb87160 2024-03-17T23:57:43.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb1463d9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55c10ca949d0 con 0x55c10cb87b50 2024-03-17T23:57:43.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb1463d9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c10cae7cf0 con 0x55c10cb8ccc0 2024-03-17T23:57:43.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb133e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb8ccc0 0x55c10cb8f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb141600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb87160 0x55c10cb87580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb133e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb8ccc0 0x55c10cb8f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:40788/0 (socket says 172.21.15.12:40788) 2024-03-17T23:57:43.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb133e00640 1 -- 172.21.15.12:0/762417846 learned_addr learned my addr 172.21.15.12:0/762417846 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb133e00640 1 -- 172.21.15.12:0/762417846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87b50 msgr2=0x55c10cb8c650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-17T23:57:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb133e00640 1 --2- 172.21.15.12:0/762417846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87b50 0x55c10cb8c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb133e00640 1 -- 172.21.15.12:0/762417846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb87160 msgr2=0x55c10cb87580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb133e00640 1 --2- 172.21.15.12:0/762417846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb87160 0x55c10cb87580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.843+0000 7fb133e00640 1 -- 172.21.15.12:0/762417846 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c10caaef10 con 0x55c10cb8ccc0 2024-03-17T23:57:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.844+0000 7fb13a000640 1 -- 172.21.15.12:0/727008331 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb12c011610 con 0x7fb13c1017f0 2024-03-17T23:57:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.844+0000 7fb133e00640 1 --2- 172.21.15.12:0/762417846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb8ccc0 0x55c10cb8f150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb12400bcb0 tx=0x7fb12400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66bfaa63c799dce6 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.844+0000 7fb131c00640 1 -- 172.21.15.12:0/762417846 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124015070 con 0x55c10cb8ccc0 2024-03-17T23:57:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.844+0000 7fb131c00640 1 -- 172.21.15.12:0/762417846 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb124005500 con 0x55c10cb8ccc0 2024-03-17T23:57:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.844+0000 7fb131c00640 1 -- 172.21.15.12:0/762417846 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1240116a0 con 0x55c10cb8ccc0 2024-03-17T23:57:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.845+0000 7fb1463d9740 1 -- 172.21.15.12:0/762417846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb8ccc0 msgr2=0x55c10cb8f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.845+0000 7fb1463d9740 1 --2- 172.21.15.12:0/762417846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb8ccc0 0x55c10cb8f150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb12400bcb0 tx=0x7fb12400f730 comp rx=0 tx=0).stop 2024-03-17T23:57:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.845+0000 7fb1463d9740 1 -- 172.21.15.12:0/762417846 shutdown_connections 2024-03-17T23:57:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.845+0000 7fb1463d9740 1 --2- 172.21.15.12:0/762417846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb8ccc0 0x55c10cb8f150 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.845+0000 7fb1463d9740 1 --2- 172.21.15.12:0/762417846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87b50 0x55c10cb8c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.845+0000 7fb1463d9740 1 --2- 172.21.15.12:0/762417846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb87160 0x55c10cb87580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.845+0000 7fb1463d9740 1 -- 172.21.15.12:0/762417846 >> 172.21.15.12:0/762417846 conn(0x55c10cb826d0 msgr2=0x55c10cb82ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.845+0000 7fb1463d9740 1 -- 172.21.15.12:0/762417846 shutdown_connections 2024-03-17T23:57:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.845+0000 7fb1463d9740 1 -- 172.21.15.12:0/762417846 wait complete. 2024-03-17T23:57:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.845+0000 7fb1463d9740 1 -- start start 2024-03-17T23:57:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.846+0000 7fb1463d9740 1 Processor -- start 2024-03-17T23:57:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.846+0000 7fb1463d9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87160 0x55c10cb912c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.846+0000 7fb13a000640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb1200a2740 0x7fb1200a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.846+0000 7fb1463d9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb87b50 0x55c10cb91800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.846+0000 7fb1463d9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb8ccc0 0x55c10cb98840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.846+0000 7fb1463d9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c10cb8fc70 con 0x55c10cb87b50 2024-03-17T23:57:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.846+0000 7fb1463d9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55c10caa7e20 con 0x55c10cb8ccc0 2024-03-17T23:57:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.846+0000 7fb1463d9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c10cb8fdf0 con 0x55c10cb87160 2024-03-17T23:57:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.846+0000 7fb1463d9740 10 client.0 _ll_register_callbacks cb 0x55c10caa77f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.846+0000 7fb13a000640 1 -- 172.21.15.12:0/727008331 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fb12c0cf3e0 con 0x7fb13c1017f0 2024-03-17T23:57:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb141600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87160 0x55c10cb912c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb141600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87160 0x55c10cb912c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:40790/0 (socket says 172.21.15.12:40790) 2024-03-17T23:57:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb141600640 1 -- 172.21.15.12:0/3046136627 learned_addr learned my addr 172.21.15.12:0/3046136627 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb140c00640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb1200a2740 0x7fb1200a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb141600640 1 --2- 172.21.15.12:0/3046136627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb8ccc0 0x55c10cb98840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb133e00640 1 --2- 172.21.15.12:0/3046136627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb87b50 0x55c10cb91800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb141600640 1 -- 172.21.15.12:0/3046136627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb8ccc0 msgr2=0x55c10cb98840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb141600640 1 --2- 172.21.15.12:0/3046136627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb8ccc0 0x55c10cb98840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb141600640 1 -- 172.21.15.12:0/3046136627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb87b50 msgr2=0x55c10cb91800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb141600640 1 --2- 172.21.15.12:0/3046136627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb87b50 0x55c10cb91800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb141600640 1 -- 172.21.15.12:0/3046136627 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c10c8002d0 con 0x55c10cb87160 2024-03-17T23:57:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb141600640 1 --2- 172.21.15.12:0/3046136627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87160 0x55c10cb912c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb13402c240 tx=0x7fb134005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eb325798bf6a469 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb12a000640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-17T23:57:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.847+0000 7fb140c00640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb1200a2740 0x7fb1200a4c00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb12c097e00 tx=0x7fb12c009520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.848+0000 7fb12a000640 1 -- 172.21.15.12:0/3046136627 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13402cd00 con 0x55c10cb87160 2024-03-17T23:57:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.848+0000 7fb12a000640 1 -- 172.21.15.12:0/3046136627 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fb134003a10 con 0x55c10cb87160 2024-03-17T23:57:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.848+0000 7fb1463d9740 1 -- 172.21.15.12:0/3046136627 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb124005500 con 0x55c10cb87160 2024-03-17T23:57:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.848+0000 7fb12a000640 1 -- 172.21.15.12:0/3046136627 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb134003c10 con 0x55c10cb87160 2024-03-17T23:57:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.848+0000 7fb1463d9740 10 client.25369 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.848+0000 7fb1463d9740 1 -- 172.21.15.12:0/3046136627 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c10c8ca6b0 con 0x55c10cb87160 2024-03-17T23:57:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.848+0000 7fb12a000640 1 -- 172.21.15.12:0/3046136627 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb124005500 con 0x55c10cb87160 2024-03-17T23:57:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.848+0000 7fb12a000640 1 -- 172.21.15.12:0/3046136627 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb13404e430 con 0x55c10cb87160 2024-03-17T23:57:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.849+0000 7fb1463d9740 10 client.25369 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.849+0000 7fb1463d9740 10 client.25369 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-17T23:57:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.849+0000 7fb1463d9740 20 client.25369 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.849+0000 7fb1463d9740 1 --2- 172.21.15.12:0/3046136627 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55c10cb9e0b0 0x55c10cb9f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.849+0000 7fb1463d9740 4 client.25369 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:57:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.849+0000 7fb1463d9740 1 -- 172.21.15.12:0/3046136627 --> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c10cb492b0 con 0x55c10cb9e0b0 2024-03-17T23:57:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.849+0000 7fb133e00640 1 --2- 172.21.15.12:0/3046136627 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55c10cb9e0b0 0x55c10cb9f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.850+0000 7fb133e00640 1 --2- 172.21.15.12:0/3046136627 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55c10cb9e0b0 0x55c10cb9f520 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3b812906cb55315c server_cookie=427c628d5e63ee8d in_seq=0 out_seq=0 2024-03-17T23:57:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.850+0000 7fb12a000640 10 client.25369 ms_handle_connect on v2:172.21.15.191:6838/4148729123 2024-03-17T23:57:43.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.850+0000 7fb12a000640 1 -- 172.21.15.12:0/3046136627 <== mds.3 v2:172.21.15.191:6838/4148729123 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c10cb492b0 con 0x55c10cb9e0b0 2024-03-17T23:57:43.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.850+0000 7fb12a000640 10 client.25369 handle_command_reply: tid=0 2024-03-17T23:57:43.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.850+0000 7fb1463d9740 1 client.25369 shutdown 2024-03-17T23:57:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.851+0000 7fb1463d9740 1 -- 172.21.15.12:0/3046136627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87160 msgr2=0x55c10cb912c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.851+0000 7fb1463d9740 1 --2- 172.21.15.12:0/3046136627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87160 0x55c10cb912c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb13402c240 tx=0x7fb134005c00 comp rx=0 tx=0).stop 2024-03-17T23:57:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.851+0000 7fb1463d9740 1 -- 172.21.15.12:0/3046136627 shutdown_connections 2024-03-17T23:57:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.851+0000 7fb1463d9740 1 --2- 172.21.15.12:0/3046136627 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55c10cb9e0b0 0x55c10cb9f520 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.851+0000 7fb1463d9740 1 --2- 172.21.15.12:0/3046136627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb8ccc0 0x55c10cb98840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.851+0000 7fb1463d9740 1 --2- 172.21.15.12:0/3046136627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb87b50 0x55c10cb91800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.851+0000 7fb1463d9740 1 --2- 172.21.15.12:0/3046136627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87160 0x55c10cb912c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.851+0000 7fb1463d9740 1 -- 172.21.15.12:0/3046136627 >> 172.21.15.12:0/3046136627 conn(0x55c10cb826d0 msgr2=0x55c10cb81a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.851+0000 7fb12a000640 0 client.25369 ms_handle_reset on v2:172.21.15.191:6838/4148729123 2024-03-17T23:57:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.851+0000 7fb1463d9740 1 -- 172.21.15.12:0/3046136627 shutdown_connections 2024-03-17T23:57:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.851+0000 7fb1463d9740 1 -- 172.21.15.12:0/3046136627 wait complete. 2024-03-17T23:57:43.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.851+0000 7fb1463d9740 20 client.25369 trim_cache size 0 max 16384 2024-03-17T23:57:43.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.873+0000 7fb1463d9740 1 Processor -- start 2024-03-17T23:57:43.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.873+0000 7fb1463d9740 1 -- start start 2024-03-17T23:57:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb1463d9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87160 0x55c10cb82ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb1463d9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87b50 0x55c10cb987d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb1463d9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb8ccc0 0x55c10cb915e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb1463d9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c10cb9d860 con 0x55c10cb8ccc0 2024-03-17T23:57:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb1463d9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55c10cb9df30 con 0x55c10cb87160 2024-03-17T23:57:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb1463d9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c10cb9e600 con 0x55c10cb87b50 2024-03-17T23:57:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb141600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87160 0x55c10cb82ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb141600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87160 0x55c10cb82ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:35976/0 (socket says 172.21.15.12:35976) 2024-03-17T23:57:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb141600640 1 -- 172.21.15.12:0/2111440517 learned_addr learned my addr 172.21.15.12:0/2111440517 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb133e00640 1 --2- 172.21.15.12:0/2111440517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87b50 0x55c10cb987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb141600640 1 --2- 172.21.15.12:0/2111440517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb8ccc0 0x55c10cb915e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb141600640 1 -- 172.21.15.12:0/2111440517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87b50 msgr2=0x55c10cb987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb141600640 1 --2- 172.21.15.12:0/2111440517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87b50 0x55c10cb987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb141600640 1 -- 172.21.15.12:0/2111440517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb8ccc0 msgr2=0x55c10cb915e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb141600640 1 --2- 172.21.15.12:0/2111440517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb8ccc0 0x55c10cb915e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.874+0000 7fb141600640 1 -- 172.21.15.12:0/2111440517 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c10cb58750 con 0x55c10cb87160 2024-03-17T23:57:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.875+0000 7fb141600640 1 --2- 172.21.15.12:0/2111440517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87160 0x55c10cb82ab0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb134004460 tx=0x7fb13402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cd2505293e72737 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.875+0000 7fb131200640 1 -- 172.21.15.12:0/2111440517 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134046070 con 0x55c10cb87160 2024-03-17T23:57:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.875+0000 7fb131200640 1 -- 172.21.15.12:0/2111440517 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb13404e430 con 0x55c10cb87160 2024-03-17T23:57:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.875+0000 7fb131200640 1 -- 172.21.15.12:0/2111440517 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13404ee60 con 0x55c10cb87160 2024-03-17T23:57:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.876+0000 7fb1463d9740 1 -- 172.21.15.12:0/2111440517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87160 msgr2=0x55c10cb82ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.876+0000 7fb1463d9740 1 --2- 172.21.15.12:0/2111440517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87160 0x55c10cb82ab0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb134004460 tx=0x7fb13402bc30 comp rx=0 tx=0).stop 2024-03-17T23:57:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.876+0000 7fb1463d9740 1 -- 172.21.15.12:0/2111440517 shutdown_connections 2024-03-17T23:57:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.876+0000 7fb1463d9740 1 --2- 172.21.15.12:0/2111440517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb8ccc0 0x55c10cb915e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.876+0000 7fb1463d9740 1 --2- 172.21.15.12:0/2111440517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87b50 0x55c10cb987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.876+0000 7fb1463d9740 1 --2- 172.21.15.12:0/2111440517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87160 0x55c10cb82ab0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.876+0000 7fb1463d9740 1 -- 172.21.15.12:0/2111440517 >> 172.21.15.12:0/2111440517 conn(0x55c10cb826d0 msgr2=0x55c10cb90ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.876+0000 7fb1463d9740 1 -- 172.21.15.12:0/2111440517 shutdown_connections 2024-03-17T23:57:43.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.876+0000 7fb1463d9740 1 -- 172.21.15.12:0/2111440517 wait complete. 2024-03-17T23:57:43.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.877+0000 7fb1463d9740 1 -- start start 2024-03-17T23:57:43.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.877+0000 7fb1463d9740 1 Processor -- start 2024-03-17T23:57:43.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.877+0000 7fb1463d9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87160 0x55c10cb94990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.877+0000 7fb1463d9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87b50 0x55c10cb94ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.877+0000 7fb1463d9740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb8ccc0 0x55c10cb97a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.877+0000 7fb1463d9740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c10cb9f3a0 con 0x55c10cb8ccc0 2024-03-17T23:57:43.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.877+0000 7fb1463d9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55c10cb9ecd0 con 0x55c10cb87b50 2024-03-17T23:57:43.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.877+0000 7fb1463d9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c10cb9fa70 con 0x55c10cb87160 2024-03-17T23:57:43.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.877+0000 7fb1463d9740 10 client.0 _ll_register_callbacks cb 0x7fb134004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:43.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.878+0000 7fb133e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87b50 0x55c10cb94ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.878+0000 7fb133e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87b50 0x55c10cb94ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:35978/0 (socket says 172.21.15.12:35978) 2024-03-17T23:57:43.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.878+0000 7fb133e00640 1 -- 172.21.15.12:0/3162903382 learned_addr learned my addr 172.21.15.12:0/3162903382 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:43.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.878+0000 7fb141600640 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87160 0x55c10cb94990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.878+0000 7fb141600640 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb8ccc0 0x55c10cb97a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.878+0000 7fb133e00640 1 -- 172.21.15.12:0/3162903382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87160 msgr2=0x55c10cb94990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.878+0000 7fb133e00640 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87160 0x55c10cb94990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.878+0000 7fb133e00640 1 -- 172.21.15.12:0/3162903382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb8ccc0 msgr2=0x55c10cb97a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.878+0000 7fb133e00640 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb8ccc0 0x55c10cb97a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.878+0000 7fb133e00640 1 -- 172.21.15.12:0/3162903382 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c10cb98050 con 0x55c10cb87b50 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.878+0000 7fb141600640 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87160 0x55c10cb94990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.878+0000 7fb133e00640 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87b50 0x55c10cb94ed0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb12400bcb0 tx=0x7fb12400b350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=945a09d1750f6ec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.878+0000 7fb12a000640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.879+0000 7fb12a000640 1 -- 172.21.15.12:0/3162903382 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124015040 con 0x55c10cb87b50 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.879+0000 7fb12a000640 1 -- 172.21.15.12:0/3162903382 <== mon.2 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fb124028130 con 0x55c10cb87b50 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.879+0000 7fb1463d9740 1 -- 172.21.15.12:0/3162903382 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb13404e430 con 0x55c10cb87b50 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.879+0000 7fb12a000640 1 -- 172.21.15.12:0/3162903382 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb124039050 con 0x55c10cb87b50 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.879+0000 7fb1463d9740 10 client.25391 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:43.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.879+0000 7fb1463d9740 1 -- 172.21.15.12:0/3162903382 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c10cc04340 con 0x55c10cb87b50 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.879+0000 7fb12a000640 1 -- 172.21.15.12:0/3162903382 <== mon.2 v2:172.21.15.79:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb13404e430 con 0x55c10cb87b50 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.879+0000 7fb12a000640 1 -- 172.21.15.12:0/3162903382 <== mon.2 v2:172.21.15.79:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb124005120 con 0x55c10cb87b50 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.879+0000 7fb1463d9740 10 client.25391 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.879+0000 7fb1463d9740 10 client.25391 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.880+0000 7fb1463d9740 20 client.25391 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.880+0000 7fb1463d9740 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55c10cc08c50 0x55c10cc07960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.880+0000 7fb1463d9740 4 client.25391 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.880+0000 7fb1463d9740 1 -- 172.21.15.12:0/3162903382 --> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] -- command(tid 0: {"prefix": "status"}) -- 0x55c10caf9190 con 0x55c10cc08c50 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.880+0000 7fb141600640 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55c10cc08c50 0x55c10cc07960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.880+0000 7fb141600640 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55c10cc08c50 0x55c10cc07960 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ab3c17ea3f9e6479 server_cookie=b41d50ba10cb49b5 in_seq=0 out_seq=0 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.880+0000 7fb12a000640 10 client.25391 ms_handle_connect on v2:172.21.15.191:6838/4148729123 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.880+0000 7fb12a000640 1 -- 172.21.15.12:0/3162903382 <== mds.3 v2:172.21.15.191:6838/4148729123 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55c10caf9190 con 0x55c10cc08c50 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.880+0000 7fb12a000640 10 client.25391 handle_command_reply: tid=0 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.880+0000 7fb1463d9740 1 client.25391 shutdown 2024-03-17T23:57:43.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.881+0000 7fb1463d9740 1 -- 172.21.15.12:0/3162903382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87b50 msgr2=0x55c10cb94ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.881+0000 7fb1463d9740 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87b50 0x55c10cb94ed0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb12400bcb0 tx=0x7fb12400b350 comp rx=0 tx=0).stop 2024-03-17T23:57:43.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.881+0000 7fb1463d9740 1 -- 172.21.15.12:0/3162903382 shutdown_connections 2024-03-17T23:57:43.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.881+0000 7fb1463d9740 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.191:6838/4148729123,v1:172.21.15.191:6839/4148729123] conn(0x55c10cc08c50 0x55c10cc07960 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.881+0000 7fb1463d9740 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c10cb8ccc0 0x55c10cb97a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.881+0000 7fb1463d9740 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c10cb87b50 0x55c10cb94ed0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.881+0000 7fb12a000640 0 client.25391 ms_handle_reset on v2:172.21.15.191:6838/4148729123 2024-03-17T23:57:43.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.881+0000 7fb1463d9740 1 --2- 172.21.15.12:0/3162903382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c10cb87160 0x55c10cb94990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.881+0000 7fb1463d9740 1 -- 172.21.15.12:0/3162903382 >> 172.21.15.12:0/3162903382 conn(0x55c10cb826d0 msgr2=0x55c10cb85930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.881+0000 7fb1463d9740 1 -- 172.21.15.12:0/3162903382 shutdown_connections 2024-03-17T23:57:43.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.881+0000 7fb1463d9740 1 -- 172.21.15.12:0/3162903382 wait complete. 2024-03-17T23:57:43.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.881+0000 7fb1463d9740 20 client.25391 trim_cache size 0 max 16384 2024-03-17T23:57:43.887 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-17T23:57:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:57:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 3, 2024-03-17T23:57:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24529, 2024-03-17T23:57:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-17T23:57:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-17T23:57:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-17T23:57:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 137.87973482800001, 2024-03-17T23:57:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 28, 2024-03-17T23:57:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 94, 2024-03-17T23:57:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 90, 2024-03-17T23:57:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 184.81273808 2024-03-17T23:57:43.887 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-17T23:57:43.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.882+0000 7fb143600640 1 -- 172.21.15.12:0/727008331 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb1200a2740 msgr2=0x7fb1200a4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.882+0000 7fb143600640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb1200a2740 0x7fb1200a4c00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb12c097e00 tx=0x7fb12c009520 comp rx=0 tx=0).stop 2024-03-17T23:57:43.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.882+0000 7fb143600640 1 -- 172.21.15.12:0/727008331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb13c1017f0 msgr2=0x7fb13c199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:43.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.882+0000 7fb143600640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb13c1017f0 0x7fb13c199c90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009fd0 tx=0x7fb12c00c270 comp rx=0 tx=0).stop 2024-03-17T23:57:43.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.882+0000 7fb143600640 1 -- 172.21.15.12:0/727008331 shutdown_connections 2024-03-17T23:57:43.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.882+0000 7fb143600640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb1200a2740 0x7fb1200a4c00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.882+0000 7fb143600640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb13c1033a0 0x7fb13c1a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.882+0000 7fb143600640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb13c1029e0 0x7fb13c19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.882+0000 7fb143600640 1 --2- 172.21.15.12:0/727008331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb13c1017f0 0x7fb13c199c90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:43.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.882+0000 7fb143600640 1 -- 172.21.15.12:0/727008331 >> 172.21.15.12:0/727008331 conn(0x7fb13c0fcd80 msgr2=0x7fb13c073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:43.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.882+0000 7fb143600640 1 -- 172.21.15.12:0/727008331 shutdown_connections 2024-03-17T23:57:43.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:43.882+0000 7fb143600640 1 -- 172.21.15.12:0/727008331 wait complete. 2024-03-17T23:57:43.909 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-17T23:57:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.115+0000 7f77d4000640 1 -- 172.21.15.12:0/437807881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77cc104c10 msgr2=0x7f77cc105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.115+0000 7f77d4000640 1 --2- 172.21.15.12:0/437807881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77cc104c10 0x7f77cc105090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f77b8009860 tx=0x7f77b802bb70 comp rx=0 tx=0).stop 2024-03-17T23:57:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.115+0000 7f77d4000640 1 -- 172.21.15.12:0/437807881 shutdown_connections 2024-03-17T23:57:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.115+0000 7f77d4000640 1 --2- 172.21.15.12:0/437807881 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77cc1055d0 0x7f77cc10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.115+0000 7f77d4000640 1 --2- 172.21.15.12:0/437807881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77cc104c10 0x7f77cc105090 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.115+0000 7f77d4000640 1 --2- 172.21.15.12:0/437807881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77cc103a20 0x7f77cc103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.115+0000 7f77d4000640 1 -- 172.21.15.12:0/437807881 >> 172.21.15.12:0/437807881 conn(0x7f77cc0fef90 msgr2=0x7f77cc1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.115+0000 7f77d4000640 1 -- 172.21.15.12:0/437807881 shutdown_connections 2024-03-17T23:57:44.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.115+0000 7f77d4000640 1 -- 172.21.15.12:0/437807881 wait complete. 2024-03-17T23:57:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.116+0000 7f77d4000640 1 Processor -- start 2024-03-17T23:57:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.116+0000 7f77d4000640 1 -- start start 2024-03-17T23:57:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.116+0000 7f77d4000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77cc103a20 0x7f77cc199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d4000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77cc104c10 0x7f77cc19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d4000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77cc1055d0 0x7f77cc1a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d4000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc10f6a0 con 0x7f77cc1055d0 2024-03-17T23:57:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d4000640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc10f520 con 0x7f77cc103a20 2024-03-17T23:57:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d4000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc10f820 con 0x7f77cc104c10 2024-03-17T23:57:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d1600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77cc103a20 0x7f77cc199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d1600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77cc103a20 0x7f77cc199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:35996/0 (socket says 172.21.15.12:35996) 2024-03-17T23:57:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d1600640 1 -- 172.21.15.12:0/487316534 learned_addr learned my addr 172.21.15.12:0/487316534 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d0c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77cc104c10 0x7f77cc19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d2000640 1 --2- 172.21.15.12:0/487316534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77cc1055d0 0x7f77cc1a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d1600640 1 -- 172.21.15.12:0/487316534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77cc104c10 msgr2=0x7f77cc19a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d1600640 1 --2- 172.21.15.12:0/487316534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77cc104c10 0x7f77cc19a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d1600640 1 -- 172.21.15.12:0/487316534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77cc1055d0 msgr2=0x7f77cc1a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d1600640 1 --2- 172.21.15.12:0/487316534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77cc1055d0 0x7f77cc1a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.117+0000 7f77d1600640 1 -- 172.21.15.12:0/487316534 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77cc1a1940 con 0x7f77cc103a20 2024-03-17T23:57:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.118+0000 7f77d1600640 1 --2- 172.21.15.12:0/487316534 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77cc103a20 0x7f77cc199d30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f77c000e9c0 tx=0x7f77c000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.118+0000 7f77ca000640 1 -- 172.21.15.12:0/487316534 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c000cd80 con 0x7f77cc103a20 2024-03-17T23:57:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.118+0000 7f77ca000640 1 -- 172.21.15.12:0/487316534 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77c00035f0 con 0x7f77cc103a20 2024-03-17T23:57:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.118+0000 7f77d4000640 1 -- 172.21.15.12:0/487316534 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77cc1a1c30 con 0x7f77cc103a20 2024-03-17T23:57:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.118+0000 7f77ca000640 1 -- 172.21.15.12:0/487316534 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c0010690 con 0x7f77cc103a20 2024-03-17T23:57:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.118+0000 7f77d4000640 1 -- 172.21.15.12:0/487316534 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77cc1a21d0 con 0x7f77cc103a20 2024-03-17T23:57:44.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.121+0000 7f77ca000640 1 -- 172.21.15.12:0/487316534 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f77c00037a0 con 0x7f77cc103a20 2024-03-17T23:57:44.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.121+0000 7f77d6e0b740 1 Processor -- start 2024-03-17T23:57:44.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.121+0000 7f77d6e0b740 1 -- start start 2024-03-17T23:57:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d6e0b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21160 0x55572aa21580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d6e0b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21b50 0x55572aa26650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d6e0b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 0x55572aa29150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d6e0b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55572a985c50 con 0x55572aa26cc0 2024-03-17T23:57:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d6e0b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55572a92e9d0 con 0x55572aa21b50 2024-03-17T23:57:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d6e0b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55572a981cf0 con 0x55572aa21160 2024-03-17T23:57:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d2000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21160 0x55572aa21580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d1600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21b50 0x55572aa26650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d2000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21160 0x55572aa21580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:40838/0 (socket says 172.21.15.12:40838) 2024-03-17T23:57:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d2000640 1 -- 172.21.15.12:0/2460448440 learned_addr learned my addr 172.21.15.12:0/2460448440 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d0c00640 1 --2- 172.21.15.12:0/2460448440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 0x55572aa29150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d2000640 1 -- 172.21.15.12:0/2460448440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21b50 msgr2=0x55572aa26650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d2000640 1 --2- 172.21.15.12:0/2460448440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21b50 0x55572aa26650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d2000640 1 -- 172.21.15.12:0/2460448440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 msgr2=0x55572aa29150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d2000640 1 --2- 172.21.15.12:0/2460448440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 0x55572aa29150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.122+0000 7f77d2000640 1 -- 172.21.15.12:0/2460448440 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55572a948f10 con 0x55572aa21160 2024-03-17T23:57:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.123+0000 7f77d2000640 1 --2- 172.21.15.12:0/2460448440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21160 0x55572aa21580 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f77c400e580 tx=0x7f77c40095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9db7b01e9665000 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.123+0000 7f77be600640 1 -- 172.21.15.12:0/2460448440 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c4003970 con 0x55572aa21160 2024-03-17T23:57:44.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.123+0000 7f77ca000640 1 --2- 172.21.15.12:0/487316534 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f77b40a2880 0x7f77b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.123+0000 7f77be600640 1 -- 172.21.15.12:0/2460448440 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77c400d830 con 0x55572aa21160 2024-03-17T23:57:44.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.123+0000 7f77be600640 1 -- 172.21.15.12:0/2460448440 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c40045a0 con 0x55572aa21160 2024-03-17T23:57:44.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.123+0000 7f77ca000640 1 -- 172.21.15.12:0/487316534 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f77c00cf1e0 con 0x7f77cc103a20 2024-03-17T23:57:44.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.123+0000 7f77d1600640 1 --2- 172.21.15.12:0/487316534 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f77b40a2880 0x7f77b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.124+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2460448440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21160 msgr2=0x55572aa21580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.124+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2460448440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21160 0x55572aa21580 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f77c400e580 tx=0x7f77c40095c0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.124+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2460448440 shutdown_connections 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.124+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2460448440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 0x55572aa29150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.124+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2460448440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21b50 0x55572aa26650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.124+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2460448440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21160 0x55572aa21580 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.124+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2460448440 >> 172.21.15.12:0/2460448440 conn(0x55572aa1c6d0 msgr2=0x55572aa1cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.124+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2460448440 shutdown_connections 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.124+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2460448440 wait complete. 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.124+0000 7f77d1600640 1 --2- 172.21.15.12:0/487316534 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f77b40a2880 0x7f77b40a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f77c0097b70 tx=0x7f77c00d2c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.125+0000 7f77d6e0b740 1 -- start start 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.125+0000 7f77d6e0b740 1 Processor -- start 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.125+0000 7f77d6e0b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 0x55572aa2b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.125+0000 7f77d6e0b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21b50 0x55572aa2b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.125+0000 7f77d6e0b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 0x55572aa32800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.125+0000 7f77d6e0b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55572aa29c70 con 0x55572aa26cc0 2024-03-17T23:57:44.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.125+0000 7f77d6e0b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55572a941e20 con 0x55572aa21160 2024-03-17T23:57:44.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.125+0000 7f77d6e0b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55572aa29df0 con 0x55572aa21b50 2024-03-17T23:57:44.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.126+0000 7f77d6e0b740 10 client.0 _ll_register_callbacks cb 0x55572a9417f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:44.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.126+0000 7f77d2000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 0x55572aa2b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.126+0000 7f77d0c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21b50 0x55572aa2b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.126+0000 7f77d2000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 0x55572aa2b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36018/0 (socket says 172.21.15.12:36018) 2024-03-17T23:57:44.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.126+0000 7f77d2000640 1 -- 172.21.15.12:0/2064224902 learned_addr learned my addr 172.21.15.12:0/2064224902 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.126+0000 7f77d2000640 1 --2- 172.21.15.12:0/2064224902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 0x55572aa32800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.126+0000 7f77d0c00640 1 -- 172.21.15.12:0/2064224902 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 msgr2=0x55572aa2b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.126+0000 7f77d0c00640 1 --2- 172.21.15.12:0/2064224902 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 0x55572aa2b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.126+0000 7f77d0c00640 1 -- 172.21.15.12:0/2064224902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 msgr2=0x55572aa32800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.126+0000 7f77d0c00640 1 --2- 172.21.15.12:0/2064224902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 0x55572aa32800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.126+0000 7f77d0c00640 1 -- 172.21.15.12:0/2064224902 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55572a69a2d0 con 0x55572aa21b50 2024-03-17T23:57:44.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.126+0000 7f77d2000640 1 --2- 172.21.15.12:0/2064224902 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 0x55572aa2b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:44.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.127+0000 7f77d0c00640 1 --2- 172.21.15.12:0/2064224902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21b50 0x55572aa2b7c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f77b802c0b0 tx=0x7f77b8005b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35addcac7c2ec9b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.127+0000 7f77b2a00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-17T23:57:44.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.127+0000 7f77b2a00640 1 -- 172.21.15.12:0/2064224902 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8042070 con 0x55572aa21b50 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.127+0000 7f77b2a00640 1 -- 172.21.15.12:0/2064224902 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f77b8004040 con 0x55572aa21b50 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.127+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2064224902 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77c400d830 con 0x55572aa21b50 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.128+0000 7f77b2a00640 1 -- 172.21.15.12:0/2064224902 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f77b804e050 con 0x55572aa21b50 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.128+0000 7f77d6e0b740 10 client.25387 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.128+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2064224902 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55572a7646b0 con 0x55572aa21b50 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.128+0000 7f77b2a00640 1 -- 172.21.15.12:0/2064224902 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77c400d830 con 0x55572aa21b50 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.128+0000 7f77b2a00640 1 -- 172.21.15.12:0/2064224902 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f77b8032640 con 0x55572aa21b50 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.128+0000 7f77d6e0b740 10 client.25387 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.128+0000 7f77d6e0b740 10 client.25387 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.128+0000 7f77d6e0b740 20 client.25387 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.128+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2064224902 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x55572aa38090 0x55572aa39500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.128+0000 7f77d6e0b740 4 client.25387 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.128+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2064224902 --> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55572a9e32b0 con 0x55572aa38090 2024-03-17T23:57:44.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.129+0000 7f77d2000640 1 --2- 172.21.15.12:0/2064224902 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x55572aa38090 0x55572aa39500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.129+0000 7f77d2000640 1 --2- 172.21.15.12:0/2064224902 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x55572aa38090 0x55572aa39500 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3e1b789c82a9b5be server_cookie=759edd45cb288921 in_seq=0 out_seq=0 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.129+0000 7f77b2a00640 10 client.25387 ms_handle_connect on v2:172.21.15.79:6836/3814468390 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.130+0000 7f77b2a00640 1 -- 172.21.15.12:0/2064224902 <== mds.2 v2:172.21.15.79:6836/3814468390 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55572a9e32b0 con 0x55572aa38090 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.130+0000 7f77b2a00640 10 client.25387 handle_command_reply: tid=0 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.130+0000 7f77d6e0b740 1 client.25387 shutdown 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.130+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2064224902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21b50 msgr2=0x55572aa2b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.130+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2064224902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21b50 0x55572aa2b7c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f77b802c0b0 tx=0x7f77b8005b20 comp rx=0 tx=0).stop 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.131+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2064224902 shutdown_connections 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.131+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2064224902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 0x55572aa32800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.131+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2064224902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21b50 0x55572aa2b7c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.131+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2064224902 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x55572aa38090 0x55572aa39500 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.131+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2064224902 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 0x55572aa2b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.131+0000 7f77b2a00640 0 client.25387 ms_handle_reset on v2:172.21.15.79:6836/3814468390 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.131+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2064224902 >> 172.21.15.12:0/2064224902 conn(0x55572aa1c6d0 msgr2=0x55572aa1ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.131+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2064224902 shutdown_connections 2024-03-17T23:57:44.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.131+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2064224902 wait complete. 2024-03-17T23:57:44.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.131+0000 7f77d6e0b740 20 client.25387 trim_cache size 0 max 16384 2024-03-17T23:57:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.162+0000 7f77d6e0b740 1 Processor -- start 2024-03-17T23:57:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.162+0000 7f77d6e0b740 1 -- start start 2024-03-17T23:57:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d6e0b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21160 0x55572aa20040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d6e0b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21b50 0x55572aa327c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d6e0b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 0x55572aa225f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d6e0b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55572aa37840 con 0x55572aa26cc0 2024-03-17T23:57:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d6e0b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55572aa37f10 con 0x55572aa21b50 2024-03-17T23:57:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d6e0b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55572aa385e0 con 0x55572aa21160 2024-03-17T23:57:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d0c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21b50 0x55572aa327c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d0c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21b50 0x55572aa327c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36020/0 (socket says 172.21.15.12:36020) 2024-03-17T23:57:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d0c00640 1 -- 172.21.15.12:0/4062834892 learned_addr learned my addr 172.21.15.12:0/4062834892 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d2000640 1 --2- 172.21.15.12:0/4062834892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21160 0x55572aa20040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d2000640 1 --2- 172.21.15.12:0/4062834892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 0x55572aa225f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d0c00640 1 -- 172.21.15.12:0/4062834892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21160 msgr2=0x55572aa20040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d0c00640 1 --2- 172.21.15.12:0/4062834892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21160 0x55572aa20040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d0c00640 1 -- 172.21.15.12:0/4062834892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 msgr2=0x55572aa225f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d0c00640 1 --2- 172.21.15.12:0/4062834892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 0x55572aa225f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d0c00640 1 -- 172.21.15.12:0/4062834892 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55572a9f2750 con 0x55572aa21b50 2024-03-17T23:57:44.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.163+0000 7f77d2000640 1 --2- 172.21.15.12:0/4062834892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 0x55572aa225f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:57:44.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.164+0000 7f77d0c00640 1 --2- 172.21.15.12:0/4062834892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21b50 0x55572aa327c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f77b80349e0 tx=0x7f77b802bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3e76b21ecba238e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.164+0000 7f77bdc00640 1 -- 172.21.15.12:0/4062834892 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8042040 con 0x55572aa21b50 2024-03-17T23:57:44.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.164+0000 7f77bdc00640 1 -- 172.21.15.12:0/4062834892 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77b8009450 con 0x55572aa21b50 2024-03-17T23:57:44.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.164+0000 7f77bdc00640 1 -- 172.21.15.12:0/4062834892 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8039040 con 0x55572aa21b50 2024-03-17T23:57:44.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.165+0000 7f77d6e0b740 1 -- 172.21.15.12:0/4062834892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21b50 msgr2=0x55572aa327c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.165+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/4062834892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21b50 0x55572aa327c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f77b80349e0 tx=0x7f77b802bb70 comp rx=0 tx=0).stop 2024-03-17T23:57:44.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.165+0000 7f77d6e0b740 1 -- 172.21.15.12:0/4062834892 shutdown_connections 2024-03-17T23:57:44.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.165+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/4062834892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa26cc0 0x55572aa225f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.165+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/4062834892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21b50 0x55572aa327c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.165+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/4062834892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa21160 0x55572aa20040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.165+0000 7f77d6e0b740 1 -- 172.21.15.12:0/4062834892 >> 172.21.15.12:0/4062834892 conn(0x55572aa1c6d0 msgr2=0x55572aa1f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.165+0000 7f77d6e0b740 1 -- 172.21.15.12:0/4062834892 shutdown_connections 2024-03-17T23:57:44.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.165+0000 7f77d6e0b740 1 -- 172.21.15.12:0/4062834892 wait complete. 2024-03-17T23:57:44.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.165+0000 7f77d6e0b740 1 -- start start 2024-03-17T23:57:44.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.166+0000 7f77d6e0b740 1 Processor -- start 2024-03-17T23:57:44.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.166+0000 7f77d6e0b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 0x55572aa2e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.166+0000 7f77d6e0b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa21b50 0x55572aa2e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.166+0000 7f77d6e0b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa26cc0 0x55572aa31710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.166+0000 7f77d6e0b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55572aa39380 con 0x55572aa21b50 2024-03-17T23:57:44.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.166+0000 7f77d6e0b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55572aa38cb0 con 0x55572aa21160 2024-03-17T23:57:44.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.166+0000 7f77d6e0b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55572aa39a50 con 0x55572aa26cc0 2024-03-17T23:57:44.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.166+0000 7f77d6e0b740 10 client.0 _ll_register_callbacks cb 0x7f77b80096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:44.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.167+0000 7f77d2000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 0x55572aa2e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.167+0000 7f77d2000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 0x55572aa2e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36036/0 (socket says 172.21.15.12:36036) 2024-03-17T23:57:44.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.167+0000 7f77d2000640 1 -- 172.21.15.12:0/2755703598 learned_addr learned my addr 172.21.15.12:0/2755703598 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.167+0000 7f77d2000640 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa26cc0 0x55572aa31710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.167+0000 7f77d0c00640 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa21b50 0x55572aa2e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.167+0000 7f77d2000640 1 -- 172.21.15.12:0/2755703598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa26cc0 msgr2=0x55572aa31710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.167+0000 7f77d2000640 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa26cc0 0x55572aa31710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.167+0000 7f77d2000640 1 -- 172.21.15.12:0/2755703598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa21b50 msgr2=0x55572aa2e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.167+0000 7f77d2000640 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa21b50 0x55572aa2e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:43 smithi012 ceph-mon[31541]: pgmap v408: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:44.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:43 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1382538633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:44.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.167+0000 7f77d2000640 1 -- 172.21.15.12:0/2755703598 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55572aa31d00 con 0x55572aa21160 2024-03-17T23:57:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.167+0000 7f77d0c00640 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa21b50 0x55572aa2e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:57:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.167+0000 7f77d2000640 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 0x55572aa2e200 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f77c400e580 tx=0x7f77c4026040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce8c4cc77fdbac86 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.167+0000 7f77b2a00640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-03-17T23:57:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.168+0000 7f77b2a00640 1 -- 172.21.15.12:0/2755703598 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c4038060 con 0x55572aa21160 2024-03-17T23:57:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.168+0000 7f77b2a00640 1 -- 172.21.15.12:0/2755703598 <== mon.2 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f77c402f030 con 0x55572aa21160 2024-03-17T23:57:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.168+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2755703598 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77b8009450 con 0x55572aa21160 2024-03-17T23:57:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.168+0000 7f77b2a00640 1 -- 172.21.15.12:0/2755703598 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f77c400d830 con 0x55572aa21160 2024-03-17T23:57:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.168+0000 7f77d6e0b740 10 client.25409 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.168+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2755703598 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55572aa9e0d0 con 0x55572aa21160 2024-03-17T23:57:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.168+0000 7f77b2a00640 1 -- 172.21.15.12:0/2755703598 <== mon.2 v2:172.21.15.79:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77b8009450 con 0x55572aa21160 2024-03-17T23:57:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.169+0000 7f77b2a00640 1 -- 172.21.15.12:0/2755703598 <== mon.2 v2:172.21.15.79:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f77c400b040 con 0x55572aa21160 2024-03-17T23:57:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.169+0000 7f77d6e0b740 10 client.25409 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.169+0000 7f77d6e0b740 10 client.25409 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.169+0000 7f77d6e0b740 20 client.25409 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.169+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x55572aaa29b0 0x55572aa9edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.169+0000 7f77d6e0b740 4 client.25409 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.169+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2755703598 --> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] -- command(tid 0: {"prefix": "status"}) -- 0x55572a993190 con 0x55572aaa29b0 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.169+0000 7f77d0c00640 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x55572aaa29b0 0x55572aa9edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77d0c00640 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x55572aaa29b0 0x55572aa9edf0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5b41135badb910cc server_cookie=2809b65f8b224279 in_seq=0 out_seq=0 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77b2a00640 10 client.25409 ms_handle_connect on v2:172.21.15.79:6836/3814468390 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77b2a00640 1 -- 172.21.15.12:0/2755703598 <== mds.2 v2:172.21.15.79:6836/3814468390 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55572a993190 con 0x55572aaa29b0 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77b2a00640 10 client.25409 handle_command_reply: tid=0 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77d6e0b740 1 client.25409 shutdown 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2755703598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 msgr2=0x55572aa2e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 0x55572aa2e200 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f77c400e580 tx=0x7f77c4026040 comp rx=0 tx=0).stop 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2755703598 shutdown_connections 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55572aa26cc0 0x55572aa31710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55572aa21b50 0x55572aa2e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.79:6836/3814468390,v1:172.21.15.79:6837/3814468390] conn(0x55572aaa29b0 0x55572aa9edf0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77d6e0b740 1 --2- 172.21.15.12:0/2755703598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55572aa21160 0x55572aa2e200 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2755703598 >> 172.21.15.12:0/2755703598 conn(0x55572aa1c6d0 msgr2=0x55572aa1ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77b2a00640 0 client.25409 ms_handle_reset on v2:172.21.15.79:6836/3814468390 2024-03-17T23:57:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2755703598 shutdown_connections 2024-03-17T23:57:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77d6e0b740 1 -- 172.21.15.12:0/2755703598 wait complete. 2024-03-17T23:57:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.170+0000 7f77d6e0b740 20 client.25409 trim_cache size 0 max 16384 2024-03-17T23:57:44.176 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-17T23:57:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:57:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 2, 2024-03-17T23:57:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24530, 2024-03-17T23:57:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-17T23:57:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-17T23:57:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-17T23:57:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 139.184370247, 2024-03-17T23:57:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 28, 2024-03-17T23:57:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 94, 2024-03-17T23:57:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 90, 2024-03-17T23:57:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 188.04930505499999 2024-03-17T23:57:44.177 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-17T23:57:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.171+0000 7f77d4000640 1 -- 172.21.15.12:0/487316534 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f77b40a2880 msgr2=0x7f77b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.171+0000 7f77d4000640 1 --2- 172.21.15.12:0/487316534 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f77b40a2880 0x7f77b40a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f77c0097b70 tx=0x7f77c00d2c20 comp rx=0 tx=0).stop 2024-03-17T23:57:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.171+0000 7f77d4000640 1 -- 172.21.15.12:0/487316534 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77cc103a20 msgr2=0x7f77cc199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.171+0000 7f77d4000640 1 --2- 172.21.15.12:0/487316534 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77cc103a20 0x7f77cc199d30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f77c000e9c0 tx=0x7f77c000ee80 comp rx=0 tx=0).stop 2024-03-17T23:57:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.171+0000 7f77d4000640 1 -- 172.21.15.12:0/487316534 shutdown_connections 2024-03-17T23:57:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.171+0000 7f77d4000640 1 --2- 172.21.15.12:0/487316534 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f77b40a2880 0x7f77b40a4d40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.171+0000 7f77d4000640 1 --2- 172.21.15.12:0/487316534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77cc1055d0 0x7f77cc1a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.171+0000 7f77d4000640 1 --2- 172.21.15.12:0/487316534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77cc104c10 0x7f77cc19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.171+0000 7f77d4000640 1 --2- 172.21.15.12:0/487316534 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77cc103a20 0x7f77cc199d30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.171+0000 7f77d4000640 1 -- 172.21.15.12:0/487316534 >> 172.21.15.12:0/487316534 conn(0x7f77cc0fef90 msgr2=0x7f77cc107f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.171+0000 7f77d4000640 1 -- 172.21.15.12:0/487316534 shutdown_connections 2024-03-17T23:57:44.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.171+0000 7f77d4000640 1 -- 172.21.15.12:0/487316534 wait complete. 2024-03-17T23:57:44.190 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-17T23:57:44.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:43 smithi079 ceph-mon[36241]: pgmap v408: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:44.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:43 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1382538633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:44.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:43 smithi191 ceph-mon[36708]: pgmap v408: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:44.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:43 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1382538633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-17T23:57:44.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.405+0000 7fcd95200640 1 -- 172.21.15.12:0/3266837151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd901028d0 msgr2=0x7fcd90102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.405+0000 7fcd95200640 1 --2- 172.21.15.12:0/3266837151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd901028d0 0x7fcd90102d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c009900 tx=0x7fcd7c02bc40 comp rx=0 tx=0).stop 2024-03-17T23:57:44.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.406+0000 7fcd95200640 1 -- 172.21.15.12:0/3266837151 shutdown_connections 2024-03-17T23:57:44.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.406+0000 7fcd95200640 1 --2- 172.21.15.12:0/3266837151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcd90103290 0x7fcd90109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.406+0000 7fcd95200640 1 --2- 172.21.15.12:0/3266837151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd901028d0 0x7fcd90102d50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.406+0000 7fcd95200640 1 --2- 172.21.15.12:0/3266837151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd901016e0 0x7fcd90101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.406+0000 7fcd95200640 1 -- 172.21.15.12:0/3266837151 >> 172.21.15.12:0/3266837151 conn(0x7fcd900fcc50 msgr2=0x7fcd900ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.406+0000 7fcd95200640 1 -- 172.21.15.12:0/3266837151 shutdown_connections 2024-03-17T23:57:44.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.406+0000 7fcd95200640 1 -- 172.21.15.12:0/3266837151 wait complete. 2024-03-17T23:57:44.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.407+0000 7fcd95200640 1 Processor -- start 2024-03-17T23:57:44.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.407+0000 7fcd95200640 1 -- start start 2024-03-17T23:57:44.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.408+0000 7fcd95200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd901016e0 0x7fcd90199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.408+0000 7fcd95200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd901028d0 0x7fcd9019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.408+0000 7fcd95200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcd90103290 0x7fcd901a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.408+0000 7fcd95200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9010d390 con 0x7fcd901016e0 2024-03-17T23:57:44.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.408+0000 7fcd95200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9010d210 con 0x7fcd90103290 2024-03-17T23:57:44.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.408+0000 7fcd95200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9010d510 con 0x7fcd901028d0 2024-03-17T23:57:44.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.408+0000 7fcd8f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd901016e0 0x7fcd90199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.408+0000 7fcd8ea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd901028d0 0x7fcd9019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.408+0000 7fcd8fe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcd90103290 0x7fcd901a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.408+0000 7fcd8ea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd901028d0 0x7fcd9019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:40876/0 (socket says 172.21.15.12:40876) 2024-03-17T23:57:44.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.408+0000 7fcd8ea00640 1 -- 172.21.15.12:0/1516079620 learned_addr learned my addr 172.21.15.12:0/1516079620 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.409+0000 7fcd8fe00640 1 -- 172.21.15.12:0/1516079620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd901028d0 msgr2=0x7fcd9019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.409+0000 7fcd8fe00640 1 --2- 172.21.15.12:0/1516079620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd901028d0 0x7fcd9019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.409+0000 7fcd8fe00640 1 -- 172.21.15.12:0/1516079620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd901016e0 msgr2=0x7fcd90199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.409+0000 7fcd8fe00640 1 --2- 172.21.15.12:0/1516079620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd901016e0 0x7fcd90199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.409+0000 7fcd8fe00640 1 -- 172.21.15.12:0/1516079620 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd901a1850 con 0x7fcd90103290 2024-03-17T23:57:44.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.409+0000 7fcd8fe00640 1 --2- 172.21.15.12:0/1516079620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcd90103290 0x7fcd901a11b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800ec50 tx=0x7fcd8800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.409+0000 7fcd87e00640 1 -- 172.21.15.12:0/1516079620 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd88003b70 con 0x7fcd90103290 2024-03-17T23:57:44.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.409+0000 7fcd87e00640 1 -- 172.21.15.12:0/1516079620 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd88003d10 con 0x7fcd90103290 2024-03-17T23:57:44.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.409+0000 7fcd95200640 1 -- 172.21.15.12:0/1516079620 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd901a1b40 con 0x7fcd90103290 2024-03-17T23:57:44.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.409+0000 7fcd87e00640 1 -- 172.21.15.12:0/1516079620 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd88010890 con 0x7fcd90103290 2024-03-17T23:57:44.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.410+0000 7fcd95200640 1 -- 172.21.15.12:0/1516079620 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd901a2090 con 0x7fcd90103290 2024-03-17T23:57:44.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.412+0000 7fcd87e00640 1 -- 172.21.15.12:0/1516079620 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcd88003fe0 con 0x7fcd90103290 2024-03-17T23:57:44.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.413+0000 7fcd995a1740 1 Processor -- start 2024-03-17T23:57:44.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.413+0000 7fcd995a1740 1 -- start start 2024-03-17T23:57:44.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.414+0000 7fcd995a1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d924340 0x56178d924760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.414+0000 7fcd995a1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 0x56178d929830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.414+0000 7fcd995a1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d929ea0 0x56178d92c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.414+0000 7fcd995a1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56178d888c50 con 0x56178d924d30 2024-03-17T23:57:44.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.414+0000 7fcd995a1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56178d8319d0 con 0x56178d929ea0 2024-03-17T23:57:44.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.414+0000 7fcd995a1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56178d884cf0 con 0x56178d924340 2024-03-17T23:57:44.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.414+0000 7fcd8ea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d924340 0x56178d924760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.414+0000 7fcd8ea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d924340 0x56178d924760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:40892/0 (socket says 172.21.15.12:40892) 2024-03-17T23:57:44.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.414+0000 7fcd8ea00640 1 -- 172.21.15.12:0/1683051838 learned_addr learned my addr 172.21.15.12:0/1683051838 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.414+0000 7fcd8f400640 1 --2- 172.21.15.12:0/1683051838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 0x56178d929830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.414+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/1683051838 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d929ea0 0x56178d92c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.415+0000 7fcd87e00640 1 --2- 172.21.15.12:0/1516079620 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcd740a2880 0x7fcd740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.415+0000 7fcd8f400640 1 -- 172.21.15.12:0/1683051838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d924340 msgr2=0x56178d924760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.415+0000 7fcd8f400640 1 --2- 172.21.15.12:0/1683051838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d924340 0x56178d924760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.415+0000 7fcd8f400640 1 -- 172.21.15.12:0/1683051838 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d929ea0 msgr2=0x56178d92c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.415+0000 7fcd8f400640 1 --2- 172.21.15.12:0/1683051838 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d929ea0 0x56178d92c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.415+0000 7fcd8f400640 1 -- 172.21.15.12:0/1683051838 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56178d84bf10 con 0x56178d924d30 2024-03-17T23:57:44.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.415+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/1683051838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d924340 0x56178d924760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:44.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.415+0000 7fcd87e00640 1 -- 172.21.15.12:0/1516079620 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fcd880cf3f0 con 0x7fcd90103290 2024-03-17T23:57:44.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.415+0000 7fcd8fe00640 1 --2- 172.21.15.12:0/1516079620 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcd740a2880 0x7fcd740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.415+0000 7fcd8f400640 1 --2- 172.21.15.12:0/1683051838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 0x56178d929830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcd90102730 tx=0x7fcd800107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d7c002579f30b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.416+0000 7fcd84c00640 1 -- 172.21.15.12:0/1683051838 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd80004020 con 0x56178d924d30 2024-03-17T23:57:44.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.416+0000 7fcd84c00640 1 -- 172.21.15.12:0/1683051838 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd800041c0 con 0x56178d924d30 2024-03-17T23:57:44.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.416+0000 7fcd84c00640 1 -- 172.21.15.12:0/1683051838 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd800116a0 con 0x56178d924d30 2024-03-17T23:57:44.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.416+0000 7fcd995a1740 1 -- 172.21.15.12:0/1683051838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 msgr2=0x56178d929830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.416+0000 7fcd8fe00640 1 --2- 172.21.15.12:0/1516079620 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcd740a2880 0x7fcd740a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcd880982f0 tx=0x7fcd8800ef50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.416+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1683051838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 0x56178d929830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcd90102730 tx=0x7fcd800107a0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.417+0000 7fcd995a1740 1 -- 172.21.15.12:0/1683051838 shutdown_connections 2024-03-17T23:57:44.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.417+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1683051838 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d929ea0 0x56178d92c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.417+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1683051838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 0x56178d929830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.417+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1683051838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d924340 0x56178d924760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.417+0000 7fcd995a1740 1 -- 172.21.15.12:0/1683051838 >> 172.21.15.12:0/1683051838 conn(0x56178d91f8b0 msgr2=0x56178d91fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.417+0000 7fcd995a1740 1 -- 172.21.15.12:0/1683051838 shutdown_connections 2024-03-17T23:57:44.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.417+0000 7fcd995a1740 1 -- 172.21.15.12:0/1683051838 wait complete. 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.417+0000 7fcd995a1740 1 -- start start 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.417+0000 7fcd995a1740 1 Processor -- start 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.418+0000 7fcd995a1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d92e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.418+0000 7fcd995a1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 0x56178d92e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.418+0000 7fcd995a1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d929ea0 0x56178d9359f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.418+0000 7fcd995a1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56178d92ce50 con 0x56178d924d30 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.418+0000 7fcd995a1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56178d844e20 con 0x56178d924340 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.418+0000 7fcd995a1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56178d92cfd0 con 0x56178d929ea0 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.418+0000 7fcd995a1740 10 client.0 _ll_register_callbacks cb 0x56178d8447f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.418+0000 7fcd8ea00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d92e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.418+0000 7fcd8ea00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d92e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36054/0 (socket says 172.21.15.12:36054) 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.418+0000 7fcd8ea00640 1 -- 172.21.15.12:0/1743160206 learned_addr learned my addr 172.21.15.12:0/1743160206 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.418+0000 7fcd8f400640 1 --2- 172.21.15.12:0/1743160206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 0x56178d92e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.418+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/1743160206 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d929ea0 0x56178d9359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.419+0000 7fcd8ea00640 1 -- 172.21.15.12:0/1743160206 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d929ea0 msgr2=0x56178d9359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.419+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/1743160206 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d929ea0 0x56178d9359f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.419+0000 7fcd8ea00640 1 -- 172.21.15.12:0/1743160206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 msgr2=0x56178d92e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.419+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/1743160206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 0x56178d92e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.419+0000 7fcd8ea00640 1 -- 172.21.15.12:0/1743160206 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56178d59d2d0 con 0x56178d924340 2024-03-17T23:57:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.419+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/1743160206 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d92e470 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c02efd0 tx=0x7fcd7c045040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5200452fd7b2418 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.419+0000 7fcd79600640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-03-17T23:57:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.419+0000 7fcd79600640 1 -- 172.21.15.12:0/1743160206 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c005530 con 0x56178d924340 2024-03-17T23:57:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.419+0000 7fcd79600640 1 -- 172.21.15.12:0/1743160206 <== mon.2 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fcd7c004020 con 0x56178d924340 2024-03-17T23:57:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.419+0000 7fcd995a1740 1 -- 172.21.15.12:0/1743160206 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd800041c0 con 0x56178d924340 2024-03-17T23:57:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.419+0000 7fcd79600640 1 -- 172.21.15.12:0/1743160206 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcd7c054050 con 0x56178d924340 2024-03-17T23:57:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.419+0000 7fcd995a1740 10 client.25421 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.419+0000 7fcd995a1740 1 -- 172.21.15.12:0/1743160206 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56178d6676b0 con 0x56178d924340 2024-03-17T23:57:44.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.420+0000 7fcd79600640 1 -- 172.21.15.12:0/1743160206 <== mon.2 v2:172.21.15.79:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd800041c0 con 0x56178d924340 2024-03-17T23:57:44.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.420+0000 7fcd79600640 1 -- 172.21.15.12:0/1743160206 <== mon.2 v2:172.21.15.79:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcd7c034440 con 0x56178d924340 2024-03-17T23:57:44.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.420+0000 7fcd995a1740 10 client.25421 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:44.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.420+0000 7fcd995a1740 10 client.25421 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-17T23:57:44.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.420+0000 7fcd995a1740 20 client.25421 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:44.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.420+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1743160206 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x56178d93b1b0 0x56178d93c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.420+0000 7fcd995a1740 4 client.25421 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:57:44.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.420+0000 7fcd995a1740 1 -- 172.21.15.12:0/1743160206 --> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56178d8e62b0 con 0x56178d93b1b0 2024-03-17T23:57:44.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.421+0000 7fcd8f400640 1 --2- 172.21.15.12:0/1743160206 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x56178d93b1b0 0x56178d93c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.421+0000 7fcd8f400640 1 --2- 172.21.15.12:0/1743160206 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x56178d93b1b0 0x56178d93c620 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c03049e7c3ca290b server_cookie=dbff69664ac75d48 in_seq=0 out_seq=0 2024-03-17T23:57:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.421+0000 7fcd79600640 10 client.25421 ms_handle_connect on v2:172.21.15.12:6840/2073607248 2024-03-17T23:57:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.422+0000 7fcd79600640 1 -- 172.21.15.12:0/1743160206 <== mds.1 v2:172.21.15.12:6840/2073607248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56178d8e62b0 con 0x56178d93b1b0 2024-03-17T23:57:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.422+0000 7fcd79600640 10 client.25421 handle_command_reply: tid=0 2024-03-17T23:57:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.422+0000 7fcd995a1740 1 client.25421 shutdown 2024-03-17T23:57:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.422+0000 7fcd995a1740 1 -- 172.21.15.12:0/1743160206 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 msgr2=0x56178d92e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.422+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1743160206 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d92e470 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c02efd0 tx=0x7fcd7c045040 comp rx=0 tx=0).stop 2024-03-17T23:57:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.422+0000 7fcd995a1740 1 -- 172.21.15.12:0/1743160206 shutdown_connections 2024-03-17T23:57:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.422+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1743160206 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x56178d93b1b0 0x56178d93c620 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.422+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1743160206 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d929ea0 0x56178d9359f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.422+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1743160206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 0x56178d92e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.422+0000 7fcd79600640 0 client.25421 ms_handle_reset on v2:172.21.15.12:6840/2073607248 2024-03-17T23:57:44.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.422+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1743160206 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d92e470 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.422+0000 7fcd995a1740 1 -- 172.21.15.12:0/1743160206 >> 172.21.15.12:0/1743160206 conn(0x56178d91f8b0 msgr2=0x56178d91ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.423+0000 7fcd995a1740 1 -- 172.21.15.12:0/1743160206 shutdown_connections 2024-03-17T23:57:44.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.423+0000 7fcd995a1740 1 -- 172.21.15.12:0/1743160206 wait complete. 2024-03-17T23:57:44.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.423+0000 7fcd995a1740 20 client.25421 trim_cache size 0 max 16384 2024-03-17T23:57:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.442+0000 7fcd995a1740 1 Processor -- start 2024-03-17T23:57:44.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.443+0000 7fcd995a1740 1 -- start start 2024-03-17T23:57:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.443+0000 7fcd995a1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d92e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.443+0000 7fcd995a1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 0x56178d9359b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.443+0000 7fcd995a1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d929ea0 0x56178d92e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.443+0000 7fcd995a1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56178d93a960 con 0x56178d924d30 2024-03-17T23:57:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.443+0000 7fcd995a1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56178d93b030 con 0x56178d924340 2024-03-17T23:57:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.443+0000 7fcd995a1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56178d93b700 con 0x56178d929ea0 2024-03-17T23:57:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.443+0000 7fcd8ea00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d92e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.443+0000 7fcd8f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 0x56178d9359b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.443+0000 7fcd8ea00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d92e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36062/0 (socket says 172.21.15.12:36062) 2024-03-17T23:57:44.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.443+0000 7fcd8ea00640 1 -- 172.21.15.12:0/1199495386 learned_addr learned my addr 172.21.15.12:0/1199495386 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.443+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/1199495386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d929ea0 0x56178d92e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.444+0000 7fcd8ea00640 1 -- 172.21.15.12:0/1199495386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d929ea0 msgr2=0x56178d92e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.444+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/1199495386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d929ea0 0x56178d92e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.444+0000 7fcd8ea00640 1 -- 172.21.15.12:0/1199495386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 msgr2=0x56178d9359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.444+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/1199495386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 0x56178d9359b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.444+0000 7fcd8ea00640 1 -- 172.21.15.12:0/1199495386 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56178d920620 con 0x56178d924340 2024-03-17T23:57:44.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.444+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/1199495386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d92e080 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c02e040 tx=0x7fcd7c041040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=687bb76a82a037b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.444+0000 7fcd7be00640 1 -- 172.21.15.12:0/1199495386 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c034640 con 0x56178d924340 2024-03-17T23:57:44.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.444+0000 7fcd7be00640 1 -- 172.21.15.12:0/1199495386 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd7c005330 con 0x56178d924340 2024-03-17T23:57:44.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.444+0000 7fcd7be00640 1 -- 172.21.15.12:0/1199495386 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c058910 con 0x56178d924340 2024-03-17T23:57:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.445+0000 7fcd995a1740 1 -- 172.21.15.12:0/1199495386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 msgr2=0x56178d92e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.445+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1199495386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d92e080 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c02e040 tx=0x7fcd7c041040 comp rx=0 tx=0).stop 2024-03-17T23:57:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.445+0000 7fcd995a1740 1 -- 172.21.15.12:0/1199495386 shutdown_connections 2024-03-17T23:57:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.445+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1199495386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d929ea0 0x56178d92e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.445+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1199495386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d924d30 0x56178d9359b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.445+0000 7fcd995a1740 1 --2- 172.21.15.12:0/1199495386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d92e080 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.445+0000 7fcd995a1740 1 -- 172.21.15.12:0/1199495386 >> 172.21.15.12:0/1199495386 conn(0x56178d91f8b0 msgr2=0x56178d91a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.445+0000 7fcd995a1740 1 -- 172.21.15.12:0/1199495386 shutdown_connections 2024-03-17T23:57:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.445+0000 7fcd995a1740 1 -- 172.21.15.12:0/1199495386 wait complete. 2024-03-17T23:57:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.445+0000 7fcd995a1740 1 -- start start 2024-03-17T23:57:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.446+0000 7fcd995a1740 1 Processor -- start 2024-03-17T23:57:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.446+0000 7fcd995a1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d933900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.446+0000 7fcd995a1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d924d30 0x56178d99cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.446+0000 7fcd995a1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d929ea0 0x56178d99f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.446+0000 7fcd995a1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56178d93c4a0 con 0x56178d929ea0 2024-03-17T23:57:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.446+0000 7fcd995a1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56178d93bdd0 con 0x56178d924340 2024-03-17T23:57:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.446+0000 7fcd995a1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56178d93cb70 con 0x56178d924d30 2024-03-17T23:57:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.446+0000 7fcd995a1740 10 client.0 _ll_register_callbacks cb 0x7fcd7c004b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.446+0000 7fcd8ea00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d933900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.446+0000 7fcd8f400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d924d30 0x56178d99cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.446+0000 7fcd8ea00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d933900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36078/0 (socket says 172.21.15.12:36078) 2024-03-17T23:57:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.446+0000 7fcd8ea00640 1 -- 172.21.15.12:0/67669876 learned_addr learned my addr 172.21.15.12:0/67669876 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.446+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/67669876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d929ea0 0x56178d99f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd8f400640 1 -- 172.21.15.12:0/67669876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 msgr2=0x56178d933900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd8f400640 1 --2- 172.21.15.12:0/67669876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d933900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd8f400640 1 -- 172.21.15.12:0/67669876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d929ea0 msgr2=0x56178d99f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd8f400640 1 --2- 172.21.15.12:0/67669876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d929ea0 0x56178d99f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd8f400640 1 -- 172.21.15.12:0/67669876 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56178d99fcc0 con 0x56178d924d30 2024-03-17T23:57:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/67669876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d929ea0 0x56178d99f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd8f400640 1 --2- 172.21.15.12:0/67669876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d924d30 0x56178d99cf00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x56178d920820 tx=0x7fcd8000b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27394a7ca7ade905 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd79600640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-17T23:57:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd79600640 1 -- 172.21.15.12:0/67669876 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd80005040 con 0x56178d924d30 2024-03-17T23:57:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd79600640 1 -- 172.21.15.12:0/67669876 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fcd800051e0 con 0x56178d924d30 2024-03-17T23:57:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd995a1740 1 -- 172.21.15.12:0/67669876 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd7c005330 con 0x56178d924d30 2024-03-17T23:57:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd79600640 1 -- 172.21.15.12:0/67669876 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcd800053e0 con 0x56178d924d30 2024-03-17T23:57:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd995a1740 10 client.25408 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.447+0000 7fcd995a1740 1 -- 172.21.15.12:0/67669876 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56178d9a0fc0 con 0x56178d924d30 2024-03-17T23:57:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.448+0000 7fcd79600640 1 -- 172.21.15.12:0/67669876 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd7c005330 con 0x56178d924d30 2024-03-17T23:57:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.448+0000 7fcd79600640 1 -- 172.21.15.12:0/67669876 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcd800049f0 con 0x56178d924d30 2024-03-17T23:57:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.448+0000 7fcd995a1740 10 client.25408 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:44.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.448+0000 7fcd995a1740 10 client.25408 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-17T23:57:44.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.448+0000 7fcd995a1740 20 client.25408 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:44.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.448+0000 7fcd995a1740 1 --2- 172.21.15.12:0/67669876 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x56178d9a57f0 0x56178d9a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.448+0000 7fcd995a1740 4 client.25408 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:57:44.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.448+0000 7fcd995a1740 1 -- 172.21.15.12:0/67669876 --> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] -- command(tid 0: {"prefix": "status"}) -- 0x56178d896190 con 0x56178d9a57f0 2024-03-17T23:57:44.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.448+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/67669876 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x56178d9a57f0 0x56178d9a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.449+0000 7fcd8ea00640 1 --2- 172.21.15.12:0/67669876 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x56178d9a57f0 0x56178d9a1c30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a48c7aa37360157 server_cookie=e868556711e0254d in_seq=0 out_seq=0 2024-03-17T23:57:44.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.449+0000 7fcd79600640 10 client.25408 ms_handle_connect on v2:172.21.15.12:6840/2073607248 2024-03-17T23:57:44.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.449+0000 7fcd79600640 1 -- 172.21.15.12:0/67669876 <== mds.1 v2:172.21.15.12:6840/2073607248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56178d896190 con 0x56178d9a57f0 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.449+0000 7fcd79600640 10 client.25408 handle_command_reply: tid=0 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.449+0000 7fcd995a1740 1 client.25408 shutdown 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.450+0000 7fcd995a1740 1 -- 172.21.15.12:0/67669876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d924d30 msgr2=0x56178d99cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.450+0000 7fcd995a1740 1 --2- 172.21.15.12:0/67669876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d924d30 0x56178d99cf00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x56178d920820 tx=0x7fcd8000b870 comp rx=0 tx=0).stop 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.450+0000 7fcd995a1740 1 -- 172.21.15.12:0/67669876 shutdown_connections 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.450+0000 7fcd995a1740 1 --2- 172.21.15.12:0/67669876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56178d929ea0 0x56178d99f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.450+0000 7fcd995a1740 1 --2- 172.21.15.12:0/67669876 >> [v2:172.21.15.12:6840/2073607248,v1:172.21.15.12:6841/2073607248] conn(0x56178d9a57f0 0x56178d9a1c30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.450+0000 7fcd995a1740 1 --2- 172.21.15.12:0/67669876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56178d924d30 0x56178d99cf00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.450+0000 7fcd995a1740 1 --2- 172.21.15.12:0/67669876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56178d924340 0x56178d933900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.450+0000 7fcd995a1740 1 -- 172.21.15.12:0/67669876 >> 172.21.15.12:0/67669876 conn(0x56178d91f8b0 msgr2=0x56178d92e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.450+0000 7fcd79600640 0 client.25408 ms_handle_reset on v2:172.21.15.12:6840/2073607248 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.450+0000 7fcd995a1740 1 -- 172.21.15.12:0/67669876 shutdown_connections 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.450+0000 7fcd995a1740 1 -- 172.21.15.12:0/67669876 wait complete. 2024-03-17T23:57:44.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.450+0000 7fcd995a1740 20 client.25408 trim_cache size 0 max 16384 2024-03-17T23:57:44.457 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-17T23:57:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:57:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 1, 2024-03-17T23:57:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24575, 2024-03-17T23:57:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-17T23:57:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-17T23:57:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-17T23:57:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 140.47674719599999, 2024-03-17T23:57:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 28, 2024-03-17T23:57:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 94, 2024-03-17T23:57:44.458 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 90, 2024-03-17T23:57:44.458 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 177.04739779100001 2024-03-17T23:57:44.458 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-17T23:57:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.451+0000 7fcd95200640 1 -- 172.21.15.12:0/1516079620 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcd740a2880 msgr2=0x7fcd740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.451+0000 7fcd95200640 1 --2- 172.21.15.12:0/1516079620 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcd740a2880 0x7fcd740a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcd880982f0 tx=0x7fcd8800ef50 comp rx=0 tx=0).stop 2024-03-17T23:57:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.451+0000 7fcd95200640 1 -- 172.21.15.12:0/1516079620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcd90103290 msgr2=0x7fcd901a11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.451+0000 7fcd95200640 1 --2- 172.21.15.12:0/1516079620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcd90103290 0x7fcd901a11b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800ec50 tx=0x7fcd8800c6a0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.451+0000 7fcd95200640 1 -- 172.21.15.12:0/1516079620 shutdown_connections 2024-03-17T23:57:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.451+0000 7fcd95200640 1 --2- 172.21.15.12:0/1516079620 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fcd740a2880 0x7fcd740a4d40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.451+0000 7fcd95200640 1 --2- 172.21.15.12:0/1516079620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcd90103290 0x7fcd901a11b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.451+0000 7fcd95200640 1 --2- 172.21.15.12:0/1516079620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd901028d0 0x7fcd9019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.451+0000 7fcd95200640 1 --2- 172.21.15.12:0/1516079620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd901016e0 0x7fcd90199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.451+0000 7fcd95200640 1 -- 172.21.15.12:0/1516079620 >> 172.21.15.12:0/1516079620 conn(0x7fcd900fcc50 msgr2=0x7fcd90073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.451+0000 7fcd95200640 1 -- 172.21.15.12:0/1516079620 shutdown_connections 2024-03-17T23:57:44.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.451+0000 7fcd95200640 1 -- 172.21.15.12:0/1516079620 wait complete. 2024-03-17T23:57:44.469 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-17T23:57:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.680+0000 7f6765200640 1 -- 172.21.15.12:0/28978271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67601022e0 msgr2=0x7f67601046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.680+0000 7f6765200640 1 --2- 172.21.15.12:0/28978271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67601022e0 0x7f67601046d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6750009900 tx=0x7f675002bc30 comp rx=0 tx=0).stop 2024-03-17T23:57:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.680+0000 7f6765200640 1 -- 172.21.15.12:0/28978271 shutdown_connections 2024-03-17T23:57:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.680+0000 7f6765200640 1 --2- 172.21.15.12:0/28978271 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6760107540 0x7f6760109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.680+0000 7f6765200640 1 --2- 172.21.15.12:0/28978271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760104c10 0x7f6760107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.680+0000 7f6765200640 1 --2- 172.21.15.12:0/28978271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67601022e0 0x7f67601046d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.680+0000 7f6765200640 1 -- 172.21.15.12:0/28978271 >> 172.21.15.12:0/28978271 conn(0x7f67600fbc90 msgr2=0x7f67600fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.680+0000 7f6765200640 1 -- 172.21.15.12:0/28978271 shutdown_connections 2024-03-17T23:57:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.680+0000 7f6765200640 1 -- 172.21.15.12:0/28978271 wait complete. 2024-03-17T23:57:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.682+0000 7f6765200640 1 Processor -- start 2024-03-17T23:57:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.682+0000 7f6765200640 1 -- start start 2024-03-17T23:57:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.682+0000 7f6765200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67601022e0 0x7f6760101c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.682+0000 7f6765200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6760104c10 0x7f6760100290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.682+0000 7f6765200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6760107540 0x7f67601007d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.682+0000 7f6765200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f676010d4c0 con 0x7f67601022e0 2024-03-17T23:57:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.682+0000 7f6765200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f676010d340 con 0x7f6760107540 2024-03-17T23:57:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.682+0000 7f6765200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f676010d640 con 0x7f6760104c10 2024-03-17T23:57:44.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.683+0000 7f675de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67601022e0 0x7f6760101c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.683+0000 7f675d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6760104c10 0x7f6760100290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.683+0000 7f675e800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6760107540 0x7f67601007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.683+0000 7f675de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67601022e0 0x7f6760101c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50178/0 (socket says 172.21.15.12:50178) 2024-03-17T23:57:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.683+0000 7f675de00640 1 -- 172.21.15.12:0/4236428621 learned_addr learned my addr 172.21.15.12:0/4236428621 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.683+0000 7f675d400640 1 -- 172.21.15.12:0/4236428621 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6760107540 msgr2=0x7f67601007d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.683+0000 7f675d400640 1 --2- 172.21.15.12:0/4236428621 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6760107540 0x7f67601007d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.683+0000 7f675d400640 1 -- 172.21.15.12:0/4236428621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67601022e0 msgr2=0x7f6760101c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.683+0000 7f675d400640 1 --2- 172.21.15.12:0/4236428621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67601022e0 0x7f6760101c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.683+0000 7f675d400640 1 -- 172.21.15.12:0/4236428621 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6760101060 con 0x7f6760104c10 2024-03-17T23:57:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.683+0000 7f675e800640 1 --2- 172.21.15.12:0/4236428621 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6760107540 0x7f67601007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.683+0000 7f675d400640 1 --2- 172.21.15.12:0/4236428621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6760104c10 0x7f6760100290 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f674c00e9c0 tx=0x7f674c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.684+0000 7f6756a00640 1 -- 172.21.15.12:0/4236428621 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c00ccc0 con 0x7f6760104c10 2024-03-17T23:57:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.684+0000 7f6756a00640 1 -- 172.21.15.12:0/4236428621 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f674c0035f0 con 0x7f6760104c10 2024-03-17T23:57:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.684+0000 7f6765200640 1 -- 172.21.15.12:0/4236428621 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67601a1cc0 con 0x7f6760104c10 2024-03-17T23:57:44.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.684+0000 7f6756a00640 1 -- 172.21.15.12:0/4236428621 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c00ce60 con 0x7f6760104c10 2024-03-17T23:57:44.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.684+0000 7f6765200640 1 -- 172.21.15.12:0/4236428621 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67601a21b0 con 0x7f6760104c10 2024-03-17T23:57:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.687+0000 7f6756a00640 1 -- 172.21.15.12:0/4236428621 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f674c0037a0 con 0x7f6760104c10 2024-03-17T23:57:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.687+0000 7f6767e52740 1 Processor -- start 2024-03-17T23:57:44.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.687+0000 7f6767e52740 1 -- start start 2024-03-17T23:57:44.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.688+0000 7f6767e52740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 0x55efd7d58580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.688+0000 7f6767e52740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d5d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.688+0000 7f6767e52740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d60150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.688+0000 7f6767e52740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55efd7cbcc50 con 0x55efd7d58160 2024-03-17T23:57:44.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.688+0000 7f6767e52740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55efd7c659d0 con 0x55efd7d58b50 2024-03-17T23:57:44.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.688+0000 7f6767e52740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55efd7cb8cf0 con 0x55efd7d5dcc0 2024-03-17T23:57:44.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.688+0000 7f675d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d60150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.688+0000 7f675d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d60150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:40966/0 (socket says 172.21.15.12:40966) 2024-03-17T23:57:44.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.688+0000 7f675d400640 1 -- 172.21.15.12:0/2324219102 learned_addr learned my addr 172.21.15.12:0/2324219102 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.688+0000 7f675de00640 1 --2- 172.21.15.12:0/2324219102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d5d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.689+0000 7f675d400640 1 -- 172.21.15.12:0/2324219102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 msgr2=0x55efd7d5d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.689+0000 7f675d400640 1 --2- 172.21.15.12:0/2324219102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d5d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.689+0000 7f675d400640 1 -- 172.21.15.12:0/2324219102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 msgr2=0x55efd7d58580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.689+0000 7f675d400640 1 --2- 172.21.15.12:0/2324219102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 0x55efd7d58580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.689+0000 7f675d400640 1 -- 172.21.15.12:0/2324219102 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efd7c7ff10 con 0x55efd7d5dcc0 2024-03-17T23:57:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.689+0000 7f6756a00640 1 --2- 172.21.15.12:0/4236428621 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f67440a2880 0x7f67440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.689+0000 7f675d400640 1 --2- 172.21.15.12:0/2324219102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d60150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f674c0d2fa0 tx=0x7f674c0d7570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e79be576b97fb776 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.689+0000 7f6756a00640 1 -- 172.21.15.12:0/4236428621 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f674c0cf090 con 0x7f6760104c10 2024-03-17T23:57:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.689+0000 7f674b400640 1 -- 172.21.15.12:0/2324219102 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c0978f0 con 0x55efd7d5dcc0 2024-03-17T23:57:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.689+0000 7f674b400640 1 -- 172.21.15.12:0/2324219102 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f674c09bd50 con 0x55efd7d5dcc0 2024-03-17T23:57:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.689+0000 7f674b400640 1 -- 172.21.15.12:0/2324219102 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c09afb0 con 0x55efd7d5dcc0 2024-03-17T23:57:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.690+0000 7f675e800640 1 --2- 172.21.15.12:0/4236428621 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f67440a2880 0x7f67440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.690+0000 7f6767e52740 1 -- 172.21.15.12:0/2324219102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 msgr2=0x55efd7d60150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.690+0000 7f6767e52740 1 --2- 172.21.15.12:0/2324219102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d60150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f674c0d2fa0 tx=0x7f674c0d7570 comp rx=0 tx=0).stop 2024-03-17T23:57:44.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.691+0000 7f6767e52740 1 -- 172.21.15.12:0/2324219102 shutdown_connections 2024-03-17T23:57:44.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.691+0000 7f6767e52740 1 --2- 172.21.15.12:0/2324219102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d60150 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.691+0000 7f6767e52740 1 --2- 172.21.15.12:0/2324219102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d5d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.691+0000 7f6767e52740 1 --2- 172.21.15.12:0/2324219102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 0x55efd7d58580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.691+0000 7f6767e52740 1 -- 172.21.15.12:0/2324219102 >> 172.21.15.12:0/2324219102 conn(0x55efd7d536d0 msgr2=0x55efd7d53ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.691+0000 7f6767e52740 1 -- 172.21.15.12:0/2324219102 shutdown_connections 2024-03-17T23:57:44.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.691+0000 7f6767e52740 1 -- 172.21.15.12:0/2324219102 wait complete. 2024-03-17T23:57:44.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.691+0000 7f675e800640 1 --2- 172.21.15.12:0/4236428621 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f67440a2880 0x7f67440a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6758009b80 tx=0x7f6758009b10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.691+0000 7f6767e52740 1 -- start start 2024-03-17T23:57:44.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f6767e52740 1 Processor -- start 2024-03-17T23:57:44.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f6767e52740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 0x55efd7d62220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f6767e52740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d62760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f6767e52740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d697a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f6767e52740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55efd7d60c10 con 0x55efd7d58160 2024-03-17T23:57:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f6767e52740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55efd7c78e20 con 0x55efd7d58b50 2024-03-17T23:57:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f6767e52740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55efd7d60d90 con 0x55efd7d5dcc0 2024-03-17T23:57:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f6767e52740 10 client.0 _ll_register_callbacks cb 0x55efd7c787f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f675e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d697a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f675de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 0x55efd7d62220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f675e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d697a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:40980/0 (socket says 172.21.15.12:40980) 2024-03-17T23:57:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f675e800640 1 -- 172.21.15.12:0/824122823 learned_addr learned my addr 172.21.15.12:0/824122823 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f675de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 0x55efd7d62220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50200/0 (socket says 172.21.15.12:50200) 2024-03-17T23:57:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.692+0000 7f675d400640 1 --2- 172.21.15.12:0/824122823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d62760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.693+0000 7f675d400640 1 -- 172.21.15.12:0/824122823 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 msgr2=0x55efd7d697a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.693+0000 7f675d400640 1 --2- 172.21.15.12:0/824122823 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d697a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.693+0000 7f675d400640 1 -- 172.21.15.12:0/824122823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 msgr2=0x55efd7d62220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.693+0000 7f675d400640 1 --2- 172.21.15.12:0/824122823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 0x55efd7d62220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.693+0000 7f675d400640 1 -- 172.21.15.12:0/824122823 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efd79d12d0 con 0x55efd7d58b50 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.693+0000 7f675e800640 1 --2- 172.21.15.12:0/824122823 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d697a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.693+0000 7f675d400640 1 --2- 172.21.15.12:0/824122823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d62760 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f674c0d2f70 tx=0x7f674c005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96d5623ccaf602e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.693+0000 7f673fe00640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.693+0000 7f673fe00640 1 -- 172.21.15.12:0/824122823 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c09b190 con 0x55efd7d58b50 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.693+0000 7f673fe00640 1 -- 172.21.15.12:0/824122823 <== mon.2 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f674c0d2740 con 0x55efd7d58b50 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.693+0000 7f6767e52740 1 -- 172.21.15.12:0/824122823 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f674c09bd50 con 0x55efd7d58b50 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.693+0000 7f673fe00640 1 -- 172.21.15.12:0/824122823 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f674c09b420 con 0x55efd7d58b50 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.694+0000 7f6767e52740 10 client.25439 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.694+0000 7f6767e52740 1 -- 172.21.15.12:0/824122823 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55efd7a9b6b0 con 0x55efd7d58b50 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.694+0000 7f673fe00640 1 -- 172.21.15.12:0/824122823 <== mon.2 v2:172.21.15.79:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f674c09bd50 con 0x55efd7d58b50 2024-03-17T23:57:44.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.694+0000 7f673fe00640 1 -- 172.21.15.12:0/824122823 <== mon.2 v2:172.21.15.79:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f674c097cc0 con 0x55efd7d58b50 2024-03-17T23:57:44.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.694+0000 7f6767e52740 10 client.25439 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:44.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.694+0000 7f6767e52740 10 client.25439 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-17T23:57:44.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.694+0000 7f6767e52740 20 client.25439 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:44.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.694+0000 7f6767e52740 1 --2- 172.21.15.12:0/824122823 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55efd7d6dc90 0x55efd7d6fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.694+0000 7f6767e52740 4 client.25439 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:57:44.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.694+0000 7f6767e52740 1 -- 172.21.15.12:0/824122823 --> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efd7d1a2b0 con 0x55efd7d6dc90 2024-03-17T23:57:44.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.694+0000 7f675de00640 1 --2- 172.21.15.12:0/824122823 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55efd7d6dc90 0x55efd7d6fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.695+0000 7f675de00640 1 --2- 172.21.15.12:0/824122823 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55efd7d6dc90 0x55efd7d6fe20 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0530a53f64cba56 server_cookie=9ffcee55ad155683 in_seq=0 out_seq=0 2024-03-17T23:57:44.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.695+0000 7f673fe00640 10 client.25439 ms_handle_connect on v2:172.21.15.79:6838/2478742711 2024-03-17T23:57:44.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.695+0000 7f673fe00640 1 -- 172.21.15.12:0/824122823 <== mds.0 v2:172.21.15.79:6838/2478742711 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55efd7d1a2b0 con 0x55efd7d6dc90 2024-03-17T23:57:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.695+0000 7f673fe00640 10 client.25439 handle_command_reply: tid=0 2024-03-17T23:57:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.696+0000 7f6767e52740 1 client.25439 shutdown 2024-03-17T23:57:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.696+0000 7f6767e52740 1 -- 172.21.15.12:0/824122823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 msgr2=0x55efd7d62760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.696+0000 7f6767e52740 1 --2- 172.21.15.12:0/824122823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d62760 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f674c0d2f70 tx=0x7f674c005b20 comp rx=0 tx=0).stop 2024-03-17T23:57:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.696+0000 7f6767e52740 1 -- 172.21.15.12:0/824122823 shutdown_connections 2024-03-17T23:57:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.696+0000 7f6767e52740 1 --2- 172.21.15.12:0/824122823 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55efd7d6dc90 0x55efd7d6fe20 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.696+0000 7f6767e52740 1 --2- 172.21.15.12:0/824122823 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d697a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.696+0000 7f6767e52740 1 --2- 172.21.15.12:0/824122823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d62760 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.696+0000 7f6767e52740 1 --2- 172.21.15.12:0/824122823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 0x55efd7d62220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.696+0000 7f673fe00640 0 client.25439 ms_handle_reset on v2:172.21.15.79:6838/2478742711 2024-03-17T23:57:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.696+0000 7f6767e52740 1 -- 172.21.15.12:0/824122823 >> 172.21.15.12:0/824122823 conn(0x55efd7d536d0 msgr2=0x55efd7d52a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.696+0000 7f6767e52740 1 -- 172.21.15.12:0/824122823 shutdown_connections 2024-03-17T23:57:44.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.696+0000 7f6767e52740 1 -- 172.21.15.12:0/824122823 wait complete. 2024-03-17T23:57:44.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.696+0000 7f6767e52740 20 client.25439 trim_cache size 0 max 16384 2024-03-17T23:57:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.716+0000 7f6767e52740 1 Processor -- start 2024-03-17T23:57:44.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.716+0000 7f6767e52740 1 -- start start 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.716+0000 7f6767e52740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d58160 0x55efd7d53ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.716+0000 7f6767e52740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58b50 0x55efd7d69760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.716+0000 7f6767e52740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d5dcc0 0x55efd7d61840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.716+0000 7f6767e52740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55efd7d6e160 con 0x55efd7d58b50 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.716+0000 7f6767e52740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55efd7d6e830 con 0x55efd7d5dcc0 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.716+0000 7f6767e52740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55efd7d6ef00 con 0x55efd7d58160 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f675d400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58b50 0x55efd7d69760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f675de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d58160 0x55efd7d53ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f675de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d58160 0x55efd7d53ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:40984/0 (socket says 172.21.15.12:40984) 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f675d400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58b50 0x55efd7d69760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50216/0 (socket says 172.21.15.12:50216) 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f675de00640 1 -- 172.21.15.12:0/1862116378 learned_addr learned my addr 172.21.15.12:0/1862116378 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f675e800640 1 --2- 172.21.15.12:0/1862116378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d5dcc0 0x55efd7d61840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f675de00640 1 -- 172.21.15.12:0/1862116378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d5dcc0 msgr2=0x55efd7d61840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f675de00640 1 --2- 172.21.15.12:0/1862116378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d5dcc0 0x55efd7d61840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f675de00640 1 -- 172.21.15.12:0/1862116378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58b50 msgr2=0x55efd7d69760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f675de00640 1 --2- 172.21.15.12:0/1862116378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58b50 0x55efd7d69760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f675de00640 1 -- 172.21.15.12:0/1862116378 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efd7d29750 con 0x55efd7d58160 2024-03-17T23:57:44.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f675e800640 1 --2- 172.21.15.12:0/1862116378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d5dcc0 0x55efd7d61840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:44.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f675de00640 1 --2- 172.21.15.12:0/1862116378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d58160 0x55efd7d53ab0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f675002c240 tx=0x7f675002df90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc5aa0435237ac68 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f674aa00640 1 -- 172.21.15.12:0/1862116378 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675002cb80 con 0x55efd7d58160 2024-03-17T23:57:44.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f674aa00640 1 -- 172.21.15.12:0/1862116378 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6750051030 con 0x55efd7d58160 2024-03-17T23:57:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.717+0000 7f674aa00640 1 -- 172.21.15.12:0/1862116378 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67500044e0 con 0x55efd7d58160 2024-03-17T23:57:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.718+0000 7f6767e52740 1 -- 172.21.15.12:0/1862116378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d58160 msgr2=0x55efd7d53ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.718+0000 7f6767e52740 1 --2- 172.21.15.12:0/1862116378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d58160 0x55efd7d53ab0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f675002c240 tx=0x7f675002df90 comp rx=0 tx=0).stop 2024-03-17T23:57:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.718+0000 7f6767e52740 1 -- 172.21.15.12:0/1862116378 shutdown_connections 2024-03-17T23:57:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.718+0000 7f6767e52740 1 --2- 172.21.15.12:0/1862116378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d5dcc0 0x55efd7d61840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.718+0000 7f6767e52740 1 --2- 172.21.15.12:0/1862116378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58b50 0x55efd7d69760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.718+0000 7f6767e52740 1 --2- 172.21.15.12:0/1862116378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d58160 0x55efd7d53ab0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.718+0000 7f6767e52740 1 -- 172.21.15.12:0/1862116378 >> 172.21.15.12:0/1862116378 conn(0x55efd7d536d0 msgr2=0x55efd7d571c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.718+0000 7f6767e52740 1 -- 172.21.15.12:0/1862116378 shutdown_connections 2024-03-17T23:57:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.718+0000 7f6767e52740 1 -- 172.21.15.12:0/1862116378 wait complete. 2024-03-17T23:57:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.719+0000 7f6767e52740 1 -- start start 2024-03-17T23:57:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.719+0000 7f6767e52740 1 Processor -- start 2024-03-17T23:57:44.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.719+0000 7f6767e52740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 0x55efd7d64c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.719+0000 7f6767e52740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d65150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.719+0000 7f6767e52740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d68120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.719+0000 7f6767e52740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55efd7d6fca0 con 0x55efd7d58160 2024-03-17T23:57:44.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.719+0000 7f6767e52740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55efd7d6f5d0 con 0x55efd7d58b50 2024-03-17T23:57:44.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.719+0000 7f6767e52740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55efd7d70370 con 0x55efd7d5dcc0 2024-03-17T23:57:44.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.719+0000 7f6767e52740 10 client.0 _ll_register_callbacks cb 0x7f674c0d2df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:57:44.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.719+0000 7f675d400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d65150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f675de00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 0x55efd7d64c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f675d400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d65150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36144/0 (socket says 172.21.15.12:36144) 2024-03-17T23:57:44.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f675e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d68120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f675d400640 1 -- 172.21.15.12:0/3165607856 learned_addr learned my addr 172.21.15.12:0/3165607856 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:44.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f675d400640 1 -- 172.21.15.12:0/3165607856 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 msgr2=0x55efd7d68120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f675d400640 1 --2- 172.21.15.12:0/3165607856 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d68120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f675d400640 1 -- 172.21.15.12:0/3165607856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 msgr2=0x55efd7d64c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f675d400640 1 --2- 172.21.15.12:0/3165607856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 0x55efd7d64c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f675d400640 1 -- 172.21.15.12:0/3165607856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efd7d68710 con 0x55efd7d58b50 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f675d400640 1 --2- 172.21.15.12:0/3165607856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d65150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f674c097ab0 tx=0x7f674c0cf310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb91fd29ecc279bb server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f673fe00640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f673fe00640 1 -- 172.21.15.12:0/3165607856 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c09b190 con 0x55efd7d58b50 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f673fe00640 1 -- 172.21.15.12:0/3165607856 <== mon.2 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f674c0a3b70 con 0x55efd7d58b50 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f6767e52740 1 -- 172.21.15.12:0/3165607856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6750051030 con 0x55efd7d58b50 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.720+0000 7f673fe00640 1 -- 172.21.15.12:0/3165607856 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f674c0978f0 con 0x55efd7d58b50 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.721+0000 7f6767e52740 10 client.25445 fetch_fsmap learned FSMap version 28 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.721+0000 7f6767e52740 1 -- 172.21.15.12:0/3165607856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55efd7d69a10 con 0x55efd7d58b50 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.721+0000 7f673fe00640 1 -- 172.21.15.12:0/3165607856 <== mon.2 v2:172.21.15.79:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6750051030 con 0x55efd7d58b50 2024-03-17T23:57:44.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.721+0000 7f673fe00640 1 -- 172.21.15.12:0/3165607856 <== mon.2 v2:172.21.15.79:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f674c09b8b0 con 0x55efd7d58b50 2024-03-17T23:57:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.721+0000 7f6767e52740 10 client.25445 fetch_fsmap finished waiting for FSMap version 28 2024-03-17T23:57:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.721+0000 7f6767e52740 10 client.25445 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-17T23:57:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.721+0000 7f6767e52740 20 client.25445 populate_metadata read hostname 'smithi012' 2024-03-17T23:57:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.721+0000 7f6767e52740 1 --2- 172.21.15.12:0/3165607856 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55efd7dd8e10 0x55efd7dd7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.721+0000 7f6767e52740 4 client.25445 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:57:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.721+0000 7f6767e52740 1 -- 172.21.15.12:0/3165607856 --> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] -- command(tid 0: {"prefix": "status"}) -- 0x55efd7cca190 con 0x55efd7dd8e10 2024-03-17T23:57:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.721+0000 7f675de00640 1 --2- 172.21.15.12:0/3165607856 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55efd7dd8e10 0x55efd7dd7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f675de00640 1 --2- 172.21.15.12:0/3165607856 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55efd7dd8e10 0x55efd7dd7b20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3660f6f75668c51f server_cookie=7867f245ff079dfc in_seq=0 out_seq=0 2024-03-17T23:57:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f673fe00640 10 client.25445 ms_handle_connect on v2:172.21.15.79:6838/2478742711 2024-03-17T23:57:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f673fe00640 1 -- 172.21.15.12:0/3165607856 <== mds.0 v2:172.21.15.79:6838/2478742711 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55efd7cca190 con 0x55efd7dd8e10 2024-03-17T23:57:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f673fe00640 10 client.25445 handle_command_reply: tid=0 2024-03-17T23:57:44.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f6767e52740 1 client.25445 shutdown 2024-03-17T23:57:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f6767e52740 1 -- 172.21.15.12:0/3165607856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 msgr2=0x55efd7d65150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f6767e52740 1 --2- 172.21.15.12:0/3165607856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d65150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f674c097ab0 tx=0x7f674c0cf310 comp rx=0 tx=0).stop 2024-03-17T23:57:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f6767e52740 1 -- 172.21.15.12:0/3165607856 shutdown_connections 2024-03-17T23:57:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f6767e52740 1 --2- 172.21.15.12:0/3165607856 >> [v2:172.21.15.79:6838/2478742711,v1:172.21.15.79:6839/2478742711] conn(0x55efd7dd8e10 0x55efd7dd7b20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f6767e52740 1 --2- 172.21.15.12:0/3165607856 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd7d5dcc0 0x55efd7d68120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f6767e52740 1 --2- 172.21.15.12:0/3165607856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55efd7d58b50 0x55efd7d65150 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f6767e52740 1 --2- 172.21.15.12:0/3165607856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55efd7d58160 0x55efd7d64c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f673fe00640 0 client.25445 ms_handle_reset on v2:172.21.15.79:6838/2478742711 2024-03-17T23:57:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f6767e52740 1 -- 172.21.15.12:0/3165607856 >> 172.21.15.12:0/3165607856 conn(0x55efd7d536d0 msgr2=0x55efd7d571c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f6767e52740 1 -- 172.21.15.12:0/3165607856 shutdown_connections 2024-03-17T23:57:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.722+0000 7f6767e52740 1 -- 172.21.15.12:0/3165607856 wait complete. 2024-03-17T23:57:44.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.723+0000 7f6767e52740 20 client.25445 trim_cache size 0 max 16384 2024-03-17T23:57:44.730 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-17T23:57:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "4558f3ea-e4b8-11ee-95c9-87774f69a715", 2024-03-17T23:57:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-03-17T23:57:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24587, 2024-03-17T23:57:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-03-17T23:57:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-03-17T23:57:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-03-17T23:57:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 167.413667269, 2024-03-17T23:57:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 28, 2024-03-17T23:57:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 94, 2024-03-17T23:57:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 90, 2024-03-17T23:57:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 174.43030150300001 2024-03-17T23:57:44.730 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-17T23:57:44.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.723+0000 7f6765200640 1 -- 172.21.15.12:0/4236428621 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f67440a2880 msgr2=0x7f67440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.723+0000 7f6765200640 1 --2- 172.21.15.12:0/4236428621 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f67440a2880 0x7f67440a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6758009b80 tx=0x7f6758009b10 comp rx=0 tx=0).stop 2024-03-17T23:57:44.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.723+0000 7f6765200640 1 -- 172.21.15.12:0/4236428621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6760104c10 msgr2=0x7f6760100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:44.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.723+0000 7f6765200640 1 --2- 172.21.15.12:0/4236428621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6760104c10 0x7f6760100290 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f674c00e9c0 tx=0x7f674c00edc0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.723+0000 7f6765200640 1 -- 172.21.15.12:0/4236428621 shutdown_connections 2024-03-17T23:57:44.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.723+0000 7f6765200640 1 --2- 172.21.15.12:0/4236428621 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f67440a2880 0x7f67440a4d40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.723+0000 7f6765200640 1 --2- 172.21.15.12:0/4236428621 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6760107540 0x7f67601007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.723+0000 7f6765200640 1 --2- 172.21.15.12:0/4236428621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6760104c10 0x7f6760100290 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.723+0000 7f6765200640 1 --2- 172.21.15.12:0/4236428621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67601022e0 0x7f6760101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:44.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.723+0000 7f6765200640 1 -- 172.21.15.12:0/4236428621 >> 172.21.15.12:0/4236428621 conn(0x7f67600fbc90 msgr2=0x7f6760102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:44.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.723+0000 7f6765200640 1 -- 172.21.15.12:0/4236428621 shutdown_connections 2024-03-17T23:57:44.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:44.723+0000 7f6765200640 1 -- 172.21.15.12:0/4236428621 wait complete. 2024-03-17T23:57:44.743 INFO:teuthology.run_tasks:Running task kclient... 2024-03-17T23:57:44.757 INFO:tasks.kclient:Mounting kernel clients... 2024-03-17T23:57:44.757 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-17T23:57:44.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:57:44.757 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-17T23:57:44.801 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-17T23:57:44.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:57:44.805 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:57:44.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:57:44.944 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-03-17T23:57:45.080 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-17T23:57:45.080 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-17T23:57:45.080 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-17T23:57:45.080 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-17T23:57:45.534 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-17T23:57:46.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:45 smithi012 ceph-mon[31541]: pgmap v409: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:46.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:45 smithi079 ceph-mon[36241]: pgmap v409: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:46.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:45 smithi191 ceph-mon[36708]: pgmap v409: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:47.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.890+0000 7fca4c600700 1 -- 172.21.15.12:0/2157275402 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44076800 msgr2=0x7fca44076be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:47.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.890+0000 7fca4c600700 1 --2- 172.21.15.12:0/2157275402 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44076800 0x7fca44076be0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fca3800b0a0 tx=0x7fca3800b3b0 comp rx=0 tx=0).stop 2024-03-17T23:57:47.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.891+0000 7fca4c600700 1 -- 172.21.15.12:0/2157275402 shutdown_connections 2024-03-17T23:57:47.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.891+0000 7fca4c600700 1 --2- 172.21.15.12:0/2157275402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca44113180 0x7fca44115570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:47.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.891+0000 7fca4c600700 1 --2- 172.21.15.12:0/2157275402 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44076800 0x7fca44076be0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:47.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.891+0000 7fca4c600700 1 --2- 172.21.15.12:0/2157275402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca44077120 0x7fca44112c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:47.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.891+0000 7fca4c600700 1 -- 172.21.15.12:0/2157275402 >> 172.21.15.12:0/2157275402 conn(0x7fca440fffb0 msgr2=0x7fca441023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:47.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.891+0000 7fca4c600700 1 -- 172.21.15.12:0/2157275402 shutdown_connections 2024-03-17T23:57:47.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.891+0000 7fca4c600700 1 -- 172.21.15.12:0/2157275402 wait complete. 2024-03-17T23:57:47.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.892+0000 7fca4c600700 1 Processor -- start 2024-03-17T23:57:47.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.892+0000 7fca4c600700 1 -- start start 2024-03-17T23:57:47.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.893+0000 7fca4c600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca44076800 0x7fca4419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:47.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.893+0000 7fca4c600700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44077120 0x7fca4419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:47.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.893+0000 7fca4c600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca44113180 0x7fca441a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:47.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.893+0000 7fca4c600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fca44118cc0 con 0x7fca44076800 2024-03-17T23:57:47.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.893+0000 7fca4c600700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fca44118b40 con 0x7fca44077120 2024-03-17T23:57:47.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.893+0000 7fca4c600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fca44118e40 con 0x7fca44113180 2024-03-17T23:57:47.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.893+0000 7fca49200700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44077120 0x7fca4419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:47.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.893+0000 7fca49c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca44076800 0x7fca4419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:47.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.893+0000 7fca49200700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44077120 0x7fca4419f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36156/0 (socket says 172.21.15.12:36156) 2024-03-17T23:57:47.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.893+0000 7fca49200700 1 -- 172.21.15.12:0/1364890373 learned_addr learned my addr 172.21.15.12:0/1364890373 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:47.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.893+0000 7fca4a600700 1 --2- 172.21.15.12:0/1364890373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca44113180 0x7fca441a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:47.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.894+0000 7fca49200700 1 -- 172.21.15.12:0/1364890373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca44113180 msgr2=0x7fca441a3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:47.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.894+0000 7fca49200700 1 --2- 172.21.15.12:0/1364890373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca44113180 0x7fca441a3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:47.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.894+0000 7fca49200700 1 -- 172.21.15.12:0/1364890373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca44076800 msgr2=0x7fca4419f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:47.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.894+0000 7fca49200700 1 --2- 172.21.15.12:0/1364890373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca44076800 0x7fca4419f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:47.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.894+0000 7fca49200700 1 -- 172.21.15.12:0/1364890373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca441a4180 con 0x7fca44077120 2024-03-17T23:57:47.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.894+0000 7fca49200700 1 --2- 172.21.15.12:0/1364890373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44077120 0x7fca4419f6b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fca3000e970 tx=0x7fca3000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:47.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.894+0000 7fca42000700 1 -- 172.21.15.12:0/1364890373 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3000c9c0 con 0x7fca44077120 2024-03-17T23:57:47.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.895+0000 7fca42000700 1 -- 172.21.15.12:0/1364890373 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca3000cb60 con 0x7fca44077120 2024-03-17T23:57:47.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.895+0000 7fca42000700 1 -- 172.21.15.12:0/1364890373 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca30018640 con 0x7fca44077120 2024-03-17T23:57:47.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.895+0000 7fca4c600700 1 -- 172.21.15.12:0/1364890373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca441a4410 con 0x7fca44077120 2024-03-17T23:57:47.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.895+0000 7fca4c600700 1 -- 172.21.15.12:0/1364890373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca441abcf0 con 0x7fca44077120 2024-03-17T23:57:47.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.896+0000 7fca4c600700 1 -- 172.21.15.12:0/1364890373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca44068f90 con 0x7fca44077120 2024-03-17T23:57:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.898+0000 7fca42000700 1 -- 172.21.15.12:0/1364890373 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fca30010430 con 0x7fca44077120 2024-03-17T23:57:47.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.899+0000 7fca42000700 1 --2- 172.21.15.12:0/1364890373 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fca2c0a2910 0x7fca2c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:47.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.900+0000 7fca42000700 1 -- 172.21.15.12:0/1364890373 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fca300cf060 con 0x7fca44077120 2024-03-17T23:57:47.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.900+0000 7fca49c00700 1 --2- 172.21.15.12:0/1364890373 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fca2c0a2910 0x7fca2c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:47.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.901+0000 7fca49c00700 1 --2- 172.21.15.12:0/1364890373 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fca2c0a2910 0x7fca2c0a4dd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fca38009fd0 tx=0x7fca38006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:47.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:47.903+0000 7fca42000700 1 -- 172.21.15.12:0/1364890373 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fca3009bd20 con 0x7fca44077120 2024-03-17T23:57:48.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:47 smithi012 ceph-mon[31541]: pgmap v410: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:57:48.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.202+0000 7fca4c600700 1 -- 172.21.15.12:0/1364890373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fca4402db70 con 0x7fca44077120 2024-03-17T23:57:48.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.203+0000 7fca42000700 1 -- 172.21.15.12:0/1364890373 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7fca3009b620 con 0x7fca44077120 2024-03-17T23:57:48.205 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-17T23:57:48.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.206+0000 7fca4c600700 1 -- 172.21.15.12:0/1364890373 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fca2c0a2910 msgr2=0x7fca2c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:48.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.206+0000 7fca4c600700 1 --2- 172.21.15.12:0/1364890373 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fca2c0a2910 0x7fca2c0a4dd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fca38009fd0 tx=0x7fca38006210 comp rx=0 tx=0).stop 2024-03-17T23:57:48.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.206+0000 7fca4c600700 1 -- 172.21.15.12:0/1364890373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44077120 msgr2=0x7fca4419f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:48.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.206+0000 7fca4c600700 1 --2- 172.21.15.12:0/1364890373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44077120 0x7fca4419f6b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fca3000e970 tx=0x7fca3000ec80 comp rx=0 tx=0).stop 2024-03-17T23:57:48.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.206+0000 7fca4c600700 1 -- 172.21.15.12:0/1364890373 shutdown_connections 2024-03-17T23:57:48.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.206+0000 7fca4c600700 1 --2- 172.21.15.12:0/1364890373 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fca2c0a2910 0x7fca2c0a4dd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:48.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.206+0000 7fca4c600700 1 --2- 172.21.15.12:0/1364890373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca44113180 0x7fca441a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:48.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.206+0000 7fca4c600700 1 --2- 172.21.15.12:0/1364890373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca44076800 0x7fca4419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:48.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.206+0000 7fca4c600700 1 --2- 172.21.15.12:0/1364890373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44077120 0x7fca4419f6b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:48.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.206+0000 7fca4c600700 1 -- 172.21.15.12:0/1364890373 >> 172.21.15.12:0/1364890373 conn(0x7fca440fffb0 msgr2=0x7fca44113df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:48.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.206+0000 7fca4c600700 1 -- 172.21.15.12:0/1364890373 shutdown_connections 2024-03-17T23:57:48.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:48.206+0000 7fca4c600700 1 -- 172.21.15.12:0/1364890373 wait complete. 2024-03-17T23:57:48.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:47 smithi079 ceph-mon[36241]: pgmap v410: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:57:48.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:47 smithi191 ceph-mon[36708]: pgmap v410: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:57:49.134 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-17T23:57:49.134 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-17T23:57:49.134 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-03-17T23:57:49.134 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-17T23:57:49.134 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-17T23:57:49.134 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-17T23:57:49.134 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-17T23:57:49.134 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-17T23:57:49.134 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:57:49.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:57:49.135 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-03-17T23:57:49.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:48 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1364890373' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-17T23:57:49.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:57:49.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:49.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:49.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:49.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:48 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:49.182 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:57:49.182 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:57:49.182 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:57:49.182 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:57:49.182 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-03-17T23:57:49.182 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:57:49.182 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:57:49.182 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:57:49.182 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-03-17T23:57:49.182 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:57:49.183 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:57:49.183 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-03-17T23:57:49.183 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-17T23:57:49.183 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-03-17T23:57:49.183 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-03-17T23:57:49.183 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-17T23:57:49.183 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 41462sec preferred_lft 41462sec 2024-03-17T23:57:49.183 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-03-17T23:57:49.183 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:57:49.183 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:57:49.183 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-03-17T23:57:49.183 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-03-17T23:57:49.184 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-17T23:57:49.185 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:57:49.185 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-17T23:57:49.185 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add name ceph-brx type bridge 2024-03-17T23:57:49.185 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr flush dev ceph-brx 2024-03-17T23:57:49.185 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set ceph-brx up 2024-03-17T23:57:49.185 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-17T23:57:49.185 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-17T23:57:49.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:48 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1364890373' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-17T23:57:49.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:57:49.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:49.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:49.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:49.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:48 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:49.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:57:49.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:48 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1364890373' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-17T23:57:49.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:57:49.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:49.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:49.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:49.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:48 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:49.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:57:49.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:57:49.502 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-17T23:57:49.573 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-03-17T23:57:49.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:57:49.578 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-03-17T23:57:49.620 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-03-17T23:57:49.620 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-17T23:57:49.620 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-17T23:57:49.621 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-17T23:57:49.621 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-17T23:57:49.621 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:57:49.621 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-17T23:57:49.621 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-17T23:57:49.621 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-17T23:57:49.621 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-17T23:57:49.621 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-17T23:57:49.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:57:49.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:57:49.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:57:49.914 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-17T23:57:49.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:57:49.956 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list-id 2024-03-17T23:57:49.998 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:49 smithi079 ceph-mon[36241]: pgmap v411: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:49.998 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:49 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:49.998 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:49 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:50.041 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:57:50.041 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-17T23:57:50.041 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:57:50.041 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-17T23:57:50.041 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-17T23:57:50.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:49 smithi191 ceph-mon[36708]: pgmap v411: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:50.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:49 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:50.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:49 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:50.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:49 smithi012 ceph-mon[31541]: pgmap v411: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:50.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:49 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:50.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:49 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:50.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:57:50.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:57:50.267 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-17T23:57:50.268 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:57:50.268 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-17T23:57:50.268 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-17T23:57:50.268 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-17T23:57:50.268 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-17T23:57:50.268 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-17T23:57:50.268 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-17T23:57:50.268 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-17T23:57:50.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:57:50.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:57:50.604 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:57:50.604 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-03-17T23:57:50.604 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-03-17T23:57:50.604 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set dev brx.0 master ceph-brx 2024-03-17T23:57:50.605 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-03-17T23:57:50.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:57:50.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:57:50.840 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:57:50.840 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-17T23:57:50.882 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:57:50.882 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:57:50.882 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:57:50.973 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-17T23:57:50.974 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-17T23:57:50.974 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/7f0ea4d6-29cb-4f0b-8d34-75c93382a378 2024-03-17T23:57:50.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:57:50.974 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/7f0ea4d6-29cb-4f0b-8d34-75c93382a378 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-17T23:57:51.102 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-17T23:57:51.102 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-17T23:57:51.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:51 smithi012 ceph-mon[31541]: pgmap v412: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:51.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:51.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:51.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:51.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:51 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:51.767 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-17T23:57:51.768 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-17T23:57:51.768 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.79:3300,172.21.15.191:3300" 2024-03-17T23:57:51.768 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@4558f3ea-e4b8-11ee-95c9-87774f69a715.cephfs=/volumes/qa/sv_1/7f0ea4d6-29cb-4f0b-8d34-75c93382a378 2024-03-17T23:57:51.768 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.12:3300/172.21.15.79:3300/172.21.15.191:3300" will pass to kernel 2024-03-17T23:57:51.772 INFO:teuthology.orchestra.run.smithi012.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-17T23:57:51.772 INFO:teuthology.orchestra.run.smithi012.stdout: You just mounted a file system that supports labels which does not 2024-03-17T23:57:51.772 INFO:teuthology.orchestra.run.smithi012.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-17T23:57:51.772 INFO:teuthology.orchestra.run.smithi012.stdout: applications will generate AVC messages and not be allowed access to 2024-03-17T23:57:51.772 INFO:teuthology.orchestra.run.smithi012.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-17T23:57:51.776 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-17T23:57:51.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:57:51.776 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:57:51.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:51 smithi191 ceph-mon[36708]: pgmap v412: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:51.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:51.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:51.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:51.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:51 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:51.856 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-17T23:57:51.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-17T23:57:51.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:57:51.964 INFO:teuthology.orchestra.run.smithi012.stdout:client25435 2024-03-17T23:57:51.964 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:57:51.964 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:57:52.007 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-17T23:57:52.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:51 smithi079 ceph-mon[36241]: pgmap v412: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:57:52.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:52.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:52.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:52.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:51 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:52.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-17T23:57:52.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:57:52.166 INFO:teuthology.orchestra.run.smithi012.stdout:client25435 2024-03-17T23:57:52.166 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-17T23:57:52.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-17T23:57:52.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:57:52.272 INFO:teuthology.orchestra.run.smithi012.stdout:client25435 2024-03-17T23:57:52.273 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715.client25435/status') 2024-03-17T23:57:52.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:57:52.412 INFO:teuthology.orchestra.run.smithi012.stdout:instance: client.25435 (3)172.21.15.12:0/719048912 2024-03-17T23:57:52.412 INFO:teuthology.orchestra.run.smithi012.stdout:blocklisted: false 2024-03-17T23:57:52.412 INFO:teuthology.orchestra.run.smithi012.stderr:0+1 records in 2024-03-17T23:57:52.412 INFO:teuthology.orchestra.run.smithi012.stderr:0+1 records out 2024-03-17T23:57:52.412 INFO:teuthology.orchestra.run.smithi012.stderr:70 bytes copied, 6.7748e-05 s, 1.0 MB/s 2024-03-17T23:57:52.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:57:52.427 INFO:teuthology.run_tasks:Running task exec... 2024-03-17T23:57:52.438 INFO:teuthology.task.exec:Executing custom commands... 2024-03-17T23:57:52.438 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-03-17T23:57:52.438 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-17T23:57:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.658+0000 7f8d67e00640 1 -- 172.21.15.12:0/1192515472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d68101650 msgr2=0x7f8d68101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.658+0000 7f8d67e00640 1 --2- 172.21.15.12:0/1192515472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d68101650 0x7f8d68101a70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009900 tx=0x7f8d5802bc30 comp rx=0 tx=0).stop 2024-03-17T23:57:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.658+0000 7f8d67e00640 1 -- 172.21.15.12:0/1192515472 shutdown_connections 2024-03-17T23:57:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.658+0000 7f8d67e00640 1 --2- 172.21.15.12:0/1192515472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d68103200 0x7f8d68109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.658+0000 7f8d67e00640 1 --2- 172.21.15.12:0/1192515472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d68102840 0x7f8d68102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.658+0000 7f8d67e00640 1 --2- 172.21.15.12:0/1192515472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d68101650 0x7f8d68101a70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.658+0000 7f8d67e00640 1 -- 172.21.15.12:0/1192515472 >> 172.21.15.12:0/1192515472 conn(0x7f8d680fcbc0 msgr2=0x7f8d680ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.658+0000 7f8d67e00640 1 -- 172.21.15.12:0/1192515472 shutdown_connections 2024-03-17T23:57:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.658+0000 7f8d67e00640 1 -- 172.21.15.12:0/1192515472 wait complete. 2024-03-17T23:57:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.659+0000 7f8d67e00640 1 Processor -- start 2024-03-17T23:57:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.659+0000 7f8d67e00640 1 -- start start 2024-03-17T23:57:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d67e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d68101650 0x7f8d68199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d67e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d68102840 0x7f8d6819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d67e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d68103200 0x7f8d681a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d67e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6810d2f0 con 0x7f8d68103200 2024-03-17T23:57:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d67e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6810d170 con 0x7f8d68102840 2024-03-17T23:57:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d67e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6810d470 con 0x7f8d68101650 2024-03-17T23:57:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d66a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d68101650 0x7f8d68199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d66a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d68101650 0x7f8d68199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:41048/0 (socket says 172.21.15.12:41048) 2024-03-17T23:57:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d66a00640 1 -- 172.21.15.12:0/842040396 learned_addr learned my addr 172.21.15.12:0/842040396 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d66000640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d68102840 0x7f8d6819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d67400640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d68103200 0x7f8d681a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d66a00640 1 -- 172.21.15.12:0/842040396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d68102840 msgr2=0x7f8d6819a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d66a00640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d68102840 0x7f8d6819a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.660+0000 7f8d66a00640 1 -- 172.21.15.12:0/842040396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d68103200 msgr2=0x7f8d681a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.661+0000 7f8d66a00640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d68103200 0x7f8d681a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:52.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.661+0000 7f8d66a00640 1 -- 172.21.15.12:0/842040396 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d681a1840 con 0x7f8d68101650 2024-03-17T23:57:52.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.661+0000 7f8d66000640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d68102840 0x7f8d6819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:57:52.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.661+0000 7f8d67400640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d68103200 0x7f8d681a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:57:52.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.661+0000 7f8d66a00640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d68101650 0x7f8d68199c30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8d58005ab0 tx=0x7f8d5802c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:52.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.661+0000 7f8d6ca00640 1 -- 172.21.15.12:0/842040396 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d58043020 con 0x7f8d68101650 2024-03-17T23:57:52.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.661+0000 7f8d6ca00640 1 -- 172.21.15.12:0/842040396 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d580054e0 con 0x7f8d68101650 2024-03-17T23:57:52.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.661+0000 7f8d67e00640 1 -- 172.21.15.12:0/842040396 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d681a1ad0 con 0x7f8d68101650 2024-03-17T23:57:52.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.661+0000 7f8d6ca00640 1 -- 172.21.15.12:0/842040396 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d58003ce0 con 0x7f8d68101650 2024-03-17T23:57:52.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.661+0000 7f8d67e00640 1 -- 172.21.15.12:0/842040396 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d681a2070 con 0x7f8d68101650 2024-03-17T23:57:52.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.663+0000 7f8d67e00640 1 -- 172.21.15.12:0/842040396 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d6804fd90 con 0x7f8d68101650 2024-03-17T23:57:52.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.664+0000 7f8d6ca00640 1 -- 172.21.15.12:0/842040396 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8d58046020 con 0x7f8d68101650 2024-03-17T23:57:52.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.666+0000 7f8d6ca00640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8d4c0a2870 0x7f8d4c0a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:52.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.667+0000 7f8d6ca00640 1 -- 172.21.15.12:0/842040396 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f8d580f0fe0 con 0x7f8d68101650 2024-03-17T23:57:52.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.667+0000 7f8d66000640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8d4c0a2870 0x7f8d4c0a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:52.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.668+0000 7f8d66000640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8d4c0a2870 0x7f8d4c0a4d30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8d54009fc0 tx=0x7f8d54005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:52.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.670+0000 7f8d6ca00640 1 -- 172.21.15.12:0/842040396 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8d5804d020 con 0x7f8d68101650 2024-03-17T23:57:52.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:52.922+0000 7f8d67e00640 1 -- 172.21.15.12:0/842040396 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f8d681a28d0 con 0x7f8d68101650 2024-03-17T23:57:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.127+0000 7f8d6ca00640 1 -- 172.21.15.12:0/842040396 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7f8d580bdcc0 con 0x7f8d68101650 2024-03-17T23:57:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.131+0000 7f8d67e00640 1 -- 172.21.15.12:0/842040396 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8d4c0a2870 msgr2=0x7f8d4c0a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.131+0000 7f8d67e00640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8d4c0a2870 0x7f8d4c0a4d30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8d54009fc0 tx=0x7f8d54005f90 comp rx=0 tx=0).stop 2024-03-17T23:57:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.131+0000 7f8d67e00640 1 -- 172.21.15.12:0/842040396 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d68101650 msgr2=0x7f8d68199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.131+0000 7f8d67e00640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d68101650 0x7f8d68199c30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8d58005ab0 tx=0x7f8d5802c6c0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.132+0000 7f8d67e00640 1 -- 172.21.15.12:0/842040396 shutdown_connections 2024-03-17T23:57:53.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.132+0000 7f8d67e00640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f8d4c0a2870 0x7f8d4c0a4d30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.132+0000 7f8d67e00640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d68103200 0x7f8d681a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.132+0000 7f8d67e00640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d68102840 0x7f8d6819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.132+0000 7f8d67e00640 1 --2- 172.21.15.12:0/842040396 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d68101650 0x7f8d68199c30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.132+0000 7f8d67e00640 1 -- 172.21.15.12:0/842040396 >> 172.21.15.12:0/842040396 conn(0x7f8d680fcbc0 msgr2=0x7f8d68073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:53.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.132+0000 7f8d67e00640 1 -- 172.21.15.12:0/842040396 shutdown_connections 2024-03-17T23:57:53.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.132+0000 7f8d67e00640 1 -- 172.21.15.12:0/842040396 wait complete. 2024-03-17T23:57:53.163 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-17T23:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.361+0000 7f5a47200640 1 -- 172.21.15.12:0/662328828 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a400761a0 msgr2=0x7f5a400765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.361+0000 7f5a47200640 1 --2- 172.21.15.12:0/662328828 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a400761a0 0x7f5a400765c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009860 tx=0x7f5a3002bb70 comp rx=0 tx=0).stop 2024-03-17T23:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.361+0000 7f5a47200640 1 -- 172.21.15.12:0/662328828 shutdown_connections 2024-03-17T23:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.361+0000 7f5a47200640 1 --2- 172.21.15.12:0/662328828 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a40077d80 0x7f5a40078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.361+0000 7f5a47200640 1 --2- 172.21.15.12:0/662328828 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a400773e0 0x7f5a40077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.361+0000 7f5a47200640 1 --2- 172.21.15.12:0/662328828 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a400761a0 0x7f5a400765c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.361+0000 7f5a47200640 1 -- 172.21.15.12:0/662328828 >> 172.21.15.12:0/662328828 conn(0x7f5a400fefa0 msgr2=0x7f5a401013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.361+0000 7f5a47200640 1 -- 172.21.15.12:0/662328828 shutdown_connections 2024-03-17T23:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.361+0000 7f5a47200640 1 -- 172.21.15.12:0/662328828 wait complete. 2024-03-17T23:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.363+0000 7f5a47200640 1 Processor -- start 2024-03-17T23:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.363+0000 7f5a47200640 1 -- start start 2024-03-17T23:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.363+0000 7f5a47200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a400761a0 0x7f5a4019e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.363+0000 7f5a47200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a400773e0 0x7f5a4019e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.363+0000 7f5a47200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a40077d80 0x7f5a401a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.363+0000 7f5a47200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5a40113a80 con 0x7f5a400773e0 2024-03-17T23:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.363+0000 7f5a47200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5a40113900 con 0x7f5a40077d80 2024-03-17T23:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.363+0000 7f5a47200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5a40113c00 con 0x7f5a400761a0 2024-03-17T23:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.364+0000 7f5a3f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a400773e0 0x7f5a4019e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.364+0000 7f5a45200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a40077d80 0x7f5a401a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.364+0000 7f5a45200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a40077d80 0x7f5a401a5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:36190/0 (socket says 172.21.15.12:36190) 2024-03-17T23:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.364+0000 7f5a3f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a400773e0 0x7f5a4019e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50254/0 (socket says 172.21.15.12:50254) 2024-03-17T23:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.364+0000 7f5a45200640 1 -- 172.21.15.12:0/4227940257 learned_addr learned my addr 172.21.15.12:0/4227940257 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-17T23:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.364+0000 7f5a3fe00640 1 --2- 172.21.15.12:0/4227940257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a400761a0 0x7f5a4019e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.364+0000 7f5a45200640 1 -- 172.21.15.12:0/4227940257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a400761a0 msgr2=0x7f5a4019e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.364+0000 7f5a45200640 1 --2- 172.21.15.12:0/4227940257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a400761a0 0x7f5a4019e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.364+0000 7f5a45200640 1 -- 172.21.15.12:0/4227940257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a400773e0 msgr2=0x7f5a4019e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.364+0000 7f5a45200640 1 --2- 172.21.15.12:0/4227940257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a400773e0 0x7f5a4019e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.364+0000 7f5a45200640 1 -- 172.21.15.12:0/4227940257 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a401a5c90 con 0x7f5a40077d80 2024-03-17T23:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.365+0000 7f5a45200640 1 --2- 172.21.15.12:0/4227940257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a40077d80 0x7f5a401a5620 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5a3800efc0 tx=0x7f5a3800c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.365+0000 7f5a3cc00640 1 -- 172.21.15.12:0/4227940257 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a38003970 con 0x7f5a40077d80 2024-03-17T23:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.365+0000 7f5a3cc00640 1 -- 172.21.15.12:0/4227940257 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a38004020 con 0x7f5a40077d80 2024-03-17T23:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.365+0000 7f5a47200640 1 -- 172.21.15.12:0/4227940257 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a401a5f80 con 0x7f5a40077d80 2024-03-17T23:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.365+0000 7f5a3cc00640 1 -- 172.21.15.12:0/4227940257 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3800b9f0 con 0x7f5a40077d80 2024-03-17T23:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.365+0000 7f5a47200640 1 -- 172.21.15.12:0/4227940257 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a401a6440 con 0x7f5a40077d80 2024-03-17T23:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.366+0000 7f5a47200640 1 -- 172.21.15.12:0/4227940257 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a401103a0 con 0x7f5a40077d80 2024-03-17T23:57:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.368+0000 7f5a3cc00640 1 -- 172.21.15.12:0/4227940257 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5a3801d020 con 0x7f5a40077d80 2024-03-17T23:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.369+0000 7f5a3cc00640 1 --2- 172.21.15.12:0/4227940257 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5a240a2920 0x7f5a240a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.370+0000 7f5a3cc00640 1 -- 172.21.15.12:0/4227940257 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f5a380cec40 con 0x7f5a40077d80 2024-03-17T23:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.370+0000 7f5a3fe00640 1 --2- 172.21.15.12:0/4227940257 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5a240a2920 0x7f5a240a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.371+0000 7f5a3fe00640 1 --2- 172.21.15.12:0/4227940257 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5a240a2920 0x7f5a240a4de0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009830 tx=0x7f5a30034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:57:53.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.373+0000 7f5a3cc00640 1 -- 172.21.15.12:0/4227940257 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5a3809b900 con 0x7f5a40077d80 2024-03-17T23:57:53.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:53 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:53.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:53 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:53.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:53 smithi012 ceph-mon[31541]: pgmap v413: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:53.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:53 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/842040396' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-17T23:57:53.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:53 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-17T23:57:53.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:53 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:53.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:53 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:53.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:53 smithi079 ceph-mon[36241]: pgmap v413: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:53.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:53 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/842040396' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-17T23:57:53.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:53 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-17T23:57:53.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:53 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:53.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:53 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:53.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:53 smithi191 ceph-mon[36708]: pgmap v413: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:57:53.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:53 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/842040396' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-17T23:57:53.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:53 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-17T23:57:53.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.685+0000 7f5a47200640 1 -- 172.21.15.12:0/4227940257 --> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f5a40061d80 con 0x7f5a240a2920 2024-03-17T23:57:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.710+0000 7f5a3cc00640 1 -- 172.21.15.12:0/4227940257 <== mgr.14152 v2:172.21.15.12:6800/3179836366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f5a40061d80 con 0x7f5a240a2920 2024-03-17T23:57:53.711 INFO:teuthology.orchestra.run.smithi012.stdout:{} 2024-03-17T23:57:53.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.712+0000 7f5a47200640 1 -- 172.21.15.12:0/4227940257 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5a240a2920 msgr2=0x7f5a240a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:53.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.712+0000 7f5a47200640 1 --2- 172.21.15.12:0/4227940257 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5a240a2920 0x7f5a240a4de0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009830 tx=0x7f5a30034ab0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.712+0000 7f5a47200640 1 -- 172.21.15.12:0/4227940257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a40077d80 msgr2=0x7f5a401a5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:57:53.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.712+0000 7f5a47200640 1 --2- 172.21.15.12:0/4227940257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a40077d80 0x7f5a401a5620 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5a3800efc0 tx=0x7f5a3800c800 comp rx=0 tx=0).stop 2024-03-17T23:57:53.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.713+0000 7f5a47200640 1 -- 172.21.15.12:0/4227940257 shutdown_connections 2024-03-17T23:57:53.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.713+0000 7f5a47200640 1 --2- 172.21.15.12:0/4227940257 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5a240a2920 0x7f5a240a4de0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.713+0000 7f5a47200640 1 --2- 172.21.15.12:0/4227940257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a40077d80 0x7f5a401a5620 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.713+0000 7f5a47200640 1 --2- 172.21.15.12:0/4227940257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a400773e0 0x7f5a4019e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.713+0000 7f5a47200640 1 --2- 172.21.15.12:0/4227940257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a400761a0 0x7f5a4019e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:57:53.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.713+0000 7f5a47200640 1 -- 172.21.15.12:0/4227940257 >> 172.21.15.12:0/4227940257 conn(0x7f5a400fefa0 msgr2=0x7f5a40100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:57:53.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.713+0000 7f5a47200640 1 -- 172.21.15.12:0/4227940257 shutdown_connections 2024-03-17T23:57:53.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-17T23:57:53.713+0000 7f5a47200640 1 -- 172.21.15.12:0/4227940257 wait complete. 2024-03-17T23:57:53.734 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-17T23:57:53.746 INFO:teuthology.run_tasks:Running task workunit... 2024-03-17T23:57:53.758 INFO:tasks.workunit:Pulling workunits from ref c382027d5c66ef4e0aa1131ea3e8080c658d3184 2024-03-17T23:57:53.758 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-17T23:57:53.758 INFO:tasks.workunit:timeout=3h 2024-03-17T23:57:53.758 INFO:tasks.workunit:cleanup=True 2024-03-17T23:57:53.758 DEBUG:teuthology.orchestra.run.smithi012:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-17T23:57:53.806 INFO:teuthology.orchestra.run.smithi012.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-17T23:57:53.807 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-17T23:57:53.807 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-17T23:57:53.807 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:57:53.807 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-17T23:57:53.807 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-17 23:55:28.198413373 +0000 2024-03-17T23:57:53.807 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-17 23:55:28.198413373 +0000 2024-03-17T23:57:53.807 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-17 23:57:51.848873606 +0000 2024-03-17T23:57:53.807 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-17 23:55:28.198413373 +0000 2024-03-17T23:57:53.807 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-17T23:57:53.807 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-17T23:57:53.950 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:57:53 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:57:53.696+0000 7fb910800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-17T23:57:53.950 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:57:53 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:57:53.696+0000 7fb910800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-17T23:57:53.951 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:57:53 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:57:53.696+0000 7fb910800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-17T23:57:53.951 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:57:53 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:57:53.696+0000 7fb910800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-17T23:57:53.951 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 17 23:57:53 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-17T23:57:53.696+0000 7fb910800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-17T23:57:53.955 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c382027d5c66ef4e0aa1131ea3e8080c658d3184 2024-03-17T23:57:54.015 INFO:tasks.workunit.client.0.smithi012.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-17T23:57:54.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:54 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-17T23:57:54.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:54 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:57:54.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:57:54.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:57:54.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:54.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:54 smithi012 ceph-mon[31541]: from='client.25463 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:54.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:54 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-17T23:57:54.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:54 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:57:54.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:57:54.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:57:54.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:54.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:54 smithi079 ceph-mon[36241]: from='client.25463 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:54.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:54 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-17T23:57:54.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:54 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:57:54.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:57:54.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:57:54.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:57:54.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:54 smithi191 ceph-mon[36708]: from='client.25463 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-17T23:57:55.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:55 smithi012 ceph-mon[31541]: pgmap v414: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-17T23:57:55.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:55 smithi079 ceph-mon[36241]: pgmap v414: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-17T23:57:55.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:55 smithi191 ceph-mon[36708]: pgmap v414: 129 pgs: 129 active+clean; 646 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-17T23:57:56.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:56 smithi012 ceph-mon[31541]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-17T23:57:56.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:56 smithi079 ceph-mon[36241]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-17T23:57:56.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:56 smithi191 ceph-mon[36708]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-17T23:57:57.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:57 smithi079 ceph-mon[36241]: pgmap v415: 129 pgs: 129 active+clean; 660 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-03-17T23:57:57.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:57 smithi191 ceph-mon[36708]: pgmap v415: 129 pgs: 129 active+clean; 660 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-03-17T23:57:57.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:57 smithi012 ceph-mon[31541]: pgmap v415: 129 pgs: 129 active+clean; 660 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-03-17T23:57:59.535 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:57:59 smithi012 ceph-mon[31541]: pgmap v416: 129 pgs: 129 active+clean; 660 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-03-17T23:57:59.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:57:59 smithi079 ceph-mon[36241]: pgmap v416: 129 pgs: 129 active+clean; 660 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-03-17T23:57:59.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:57:59 smithi191 ceph-mon[36708]: pgmap v416: 129 pgs: 129 active+clean; 660 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-03-17T23:58:01.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:01 smithi079 ceph-mon[36241]: pgmap v417: 129 pgs: 129 active+clean; 660 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-03-17T23:58:01.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:01 smithi191 ceph-mon[36708]: pgmap v417: 129 pgs: 129 active+clean; 660 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-03-17T23:58:01.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:01 smithi012 ceph-mon[31541]: pgmap v417: 129 pgs: 129 active+clean; 660 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-03-17T23:58:03.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:03 smithi079 ceph-mon[36241]: pgmap v418: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-17T23:58:03.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:03 smithi191 ceph-mon[36708]: pgmap v418: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-17T23:58:03.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:03 smithi012 ceph-mon[31541]: pgmap v418: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-17T23:58:05.472 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:05 smithi012 ceph-mon[31541]: pgmap v419: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-17T23:58:05.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:05 smithi079 ceph-mon[36241]: pgmap v419: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-17T23:58:05.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:05 smithi191 ceph-mon[36708]: pgmap v419: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-17T23:58:07.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:07 smithi079 ceph-mon[36241]: pgmap v420: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-03-17T23:58:07.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:07 smithi191 ceph-mon[36708]: pgmap v420: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-03-17T23:58:07.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:07 smithi012 ceph-mon[31541]: pgmap v420: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-03-17T23:58:09.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:09 smithi079 ceph-mon[36241]: pgmap v421: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-17T23:58:09.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:09 smithi191 ceph-mon[36708]: pgmap v421: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-17T23:58:09.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:09 smithi012 ceph-mon[31541]: pgmap v421: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-17T23:58:11.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:11 smithi079 ceph-mon[36241]: pgmap v422: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-17T23:58:11.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:11 smithi191 ceph-mon[36708]: pgmap v422: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-17T23:58:11.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:11 smithi012 ceph-mon[31541]: pgmap v422: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-17T23:58:13.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:13 smithi079 ceph-mon[36241]: pgmap v423: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-17T23:58:13.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:13 smithi191 ceph-mon[36708]: pgmap v423: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-17T23:58:13.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:13 smithi012 ceph-mon[31541]: pgmap v423: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-17T23:58:15.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:15 smithi079 ceph-mon[36241]: pgmap v424: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-17T23:58:15.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:15 smithi191 ceph-mon[36708]: pgmap v424: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-17T23:58:15.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:15 smithi012 ceph-mon[31541]: pgmap v424: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-17T23:58:17.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:17 smithi079 ceph-mon[36241]: pgmap v425: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-17T23:58:17.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:17 smithi191 ceph-mon[36708]: pgmap v425: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-17T23:58:17.623 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:17 smithi012 ceph-mon[31541]: pgmap v425: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-17T23:58:19.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:19 smithi079 ceph-mon[36241]: pgmap v426: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:19.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:19 smithi191 ceph-mon[36708]: pgmap v426: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:19.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:19 smithi012 ceph-mon[31541]: pgmap v426: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:21.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:21 smithi079 ceph-mon[36241]: pgmap v427: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:21.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:21 smithi191 ceph-mon[36708]: pgmap v427: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:21.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:21 smithi012 ceph-mon[31541]: pgmap v427: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:23.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:23 smithi079 ceph-mon[36241]: pgmap v428: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:58:23.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:23 smithi191 ceph-mon[36708]: pgmap v428: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:58:23.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:23 smithi012 ceph-mon[31541]: pgmap v428: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-17T23:58:25.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:25 smithi079 ceph-mon[36241]: pgmap v429: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:25.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:25 smithi191 ceph-mon[36708]: pgmap v429: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:25.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:25 smithi012 ceph-mon[31541]: pgmap v429: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:27.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:27 smithi079 ceph-mon[36241]: pgmap v430: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:58:27.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:27 smithi191 ceph-mon[36708]: pgmap v430: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:58:27.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:27 smithi012 ceph-mon[31541]: pgmap v430: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-17T23:58:29.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:29 smithi079 ceph-mon[36241]: pgmap v431: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:29.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:29 smithi191 ceph-mon[36708]: pgmap v431: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:29.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:29 smithi012 ceph-mon[31541]: pgmap v431: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:31.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:31 smithi079 ceph-mon[36241]: pgmap v432: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:31.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:31 smithi191 ceph-mon[36708]: pgmap v432: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:31.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:31 smithi012 ceph-mon[31541]: pgmap v432: 129 pgs: 129 active+clean; 662 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-17T23:58:33.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:33 smithi079 ceph-mon[36241]: pgmap v433: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-17T23:58:33.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:33 smithi191 ceph-mon[36708]: pgmap v433: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-17T23:58:33.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:33 smithi012 ceph-mon[31541]: pgmap v433: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-17T23:58:35.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:35 smithi079 ceph-mon[36241]: pgmap v434: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-17T23:58:35.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:35 smithi191 ceph-mon[36708]: pgmap v434: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-17T23:58:35.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:35 smithi012 ceph-mon[31541]: pgmap v434: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-17T23:58:37.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:37 smithi079 ceph-mon[36241]: pgmap v435: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-17T23:58:37.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:37 smithi191 ceph-mon[36708]: pgmap v435: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-17T23:58:37.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:37 smithi012 ceph-mon[31541]: pgmap v435: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-17T23:58:39.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:39 smithi079 ceph-mon[36241]: pgmap v436: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-17T23:58:39.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:39 smithi191 ceph-mon[36708]: pgmap v436: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-17T23:58:39.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:39 smithi012 ceph-mon[31541]: pgmap v436: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-17T23:58:41.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:41 smithi079 ceph-mon[36241]: pgmap v437: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-17T23:58:41.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:41 smithi191 ceph-mon[36708]: pgmap v437: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-17T23:58:41.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:41 smithi012 ceph-mon[31541]: pgmap v437: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-17T23:58:43.065 INFO:tasks.workunit.client.0.smithi012.stderr:Note: switching to 'c382027d5c66ef4e0aa1131ea3e8080c658d3184'. 2024-03-17T23:58:43.065 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-17T23:58:43.065 INFO:tasks.workunit.client.0.smithi012.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-17T23:58:43.065 INFO:tasks.workunit.client.0.smithi012.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-17T23:58:43.065 INFO:tasks.workunit.client.0.smithi012.stderr:state without impacting any branches by switching back to a branch. 2024-03-17T23:58:43.065 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-17T23:58:43.065 INFO:tasks.workunit.client.0.smithi012.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-17T23:58:43.065 INFO:tasks.workunit.client.0.smithi012.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-17T23:58:43.065 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-17T23:58:43.065 INFO:tasks.workunit.client.0.smithi012.stderr: git switch -c 2024-03-17T23:58:43.066 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-17T23:58:43.066 INFO:tasks.workunit.client.0.smithi012.stderr:Or undo this operation with: 2024-03-17T23:58:43.066 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-17T23:58:43.066 INFO:tasks.workunit.client.0.smithi012.stderr: git switch - 2024-03-17T23:58:43.066 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-17T23:58:43.066 INFO:tasks.workunit.client.0.smithi012.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-17T23:58:43.066 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-17T23:58:43.066 INFO:tasks.workunit.client.0.smithi012.stderr:HEAD is now at c382027d5c6 ignorelist mon down 2024-03-17T23:58:43.077 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-17T23:58:43.135 INFO:tasks.workunit.client.0.smithi012.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-17T23:58:43.142 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-17T23:58:43.142 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-17T23:58:43.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:43 smithi079 ceph-mon[36241]: pgmap v438: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-17T23:58:43.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:43 smithi191 ceph-mon[36708]: pgmap v438: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-17T23:58:43.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:43 smithi012 ceph-mon[31541]: pgmap v438: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-17T23:58:44.031 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-17T23:58:44.161 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-17T23:58:44.255 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-17T23:58:44.262 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-17T23:58:44.262 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-17T23:58:44.354 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-17T23:58:44.364 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-17T23:58:44.364 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-17T23:58:44.410 INFO:tasks.workunit:Running workunits matching suites/blogbench.sh on client.0... 2024-03-17T23:58:44.411 INFO:tasks.workunit:Running workunit suites/blogbench.sh... 2024-03-17T23:58:44.412 DEBUG:teuthology.orchestra.run.smithi012:workunit test suites/blogbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c382027d5c66ef4e0aa1131ea3e8080c658d3184 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh 2024-03-17T23:58:44.525 INFO:tasks.workunit.client.0.smithi012.stdout:getting blogbench 2024-03-17T23:58:44.526 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'getting blogbench' 2024-03-17T23:58:44.526 INFO:tasks.workunit.client.0.smithi012.stderr:+ wget http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-03-17T23:58:44.578 INFO:tasks.workunit.client.0.smithi012.stderr:--2024-03-17 23:58:44-- http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-03-17T23:58:44.579 INFO:tasks.workunit.client.0.smithi012.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-17T23:58:44.619 INFO:tasks.workunit.client.0.smithi012.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-17T23:58:44.655 INFO:tasks.workunit.client.0.smithi012.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-17T23:58:44.655 INFO:tasks.workunit.client.0.smithi012.stderr:Length: 103355 (101K) [application/octet-stream] 2024-03-17T23:58:44.658 INFO:tasks.workunit.client.0.smithi012.stderr:Saving to: ‘blogbench-1.0.tar.bz2’ 2024-03-17T23:58:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-17T23:58:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: 0K .......... .......... .......... .......... .......... 49% 737K 0s 2024-03-17T23:58:44.762 INFO:tasks.workunit.client.0.smithi012.stderr: 50K .......... .......... .......... .......... .......... 99% 1.38M 0s 2024-03-17T23:58:44.762 INFO:tasks.workunit.client.0.smithi012.stderr: 100K 100% 1.74T=0.1s 2024-03-17T23:58:44.762 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-17T23:58:44.762 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-17 23:58:44 (979 KB/s) - ‘blogbench-1.0.tar.bz2’ saved [103355/103355] 2024-03-17T23:58:44.762 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-17T23:58:44.764 INFO:tasks.workunit.client.0.smithi012.stderr:+ tar -xvf blogbench-1.0.tar.bz2 2024-03-17T23:58:44.805 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/ 2024-03-17T23:58:44.807 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/m4/ 2024-03-17T23:58:44.808 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/m4/acx_pthread.m4 2024-03-17T23:58:44.810 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/man/ 2024-03-17T23:58:44.812 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/man/blogbench.8 2024-03-17T23:58:44.815 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/man/Makefile.am 2024-03-17T23:58:44.816 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/man/Makefile.in 2024-03-17T23:58:44.818 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/ 2024-03-17T23:58:44.820 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/mysnprintf.c 2024-03-17T23:58:44.822 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/mysnprintf.h 2024-03-17T23:58:44.824 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/blogbench.c 2024-03-17T23:58:44.826 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/blogbench.h 2024-03-17T23:58:44.828 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/fakesnprintf.c 2024-03-17T23:58:44.830 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/fakesnprintf.h 2024-03-17T23:58:44.832 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/blogbench_p.h 2024-03-17T23:58:44.834 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/bsd-getopt_long.c 2024-03-17T23:58:44.836 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/bsd-getopt_long.h 2024-03-17T23:58:44.838 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/Makefile.am 2024-03-17T23:58:44.840 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/Makefile.in 2024-03-17T23:58:44.842 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/process.c 2024-03-17T23:58:44.844 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/globals.h 2024-03-17T23:58:44.846 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/commenter.c 2024-03-17T23:58:44.848 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/rewriter.c 2024-03-17T23:58:44.850 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/reader.c 2024-03-17T23:58:44.852 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/writer.c 2024-03-17T23:58:44.854 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/src/helpers.c 2024-03-17T23:58:44.856 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/NEWS 2024-03-17T23:58:44.858 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/compile 2024-03-17T23:58:44.860 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/depcomp 2024-03-17T23:58:44.862 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/aclocal.m4 2024-03-17T23:58:44.864 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/README 2024-03-17T23:58:44.865 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/configure 2024-03-17T23:58:44.876 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/configure.ac 2024-03-17T23:58:44.879 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/config.guess 2024-03-17T23:58:44.881 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/install-sh 2024-03-17T23:58:44.883 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/config.sub 2024-03-17T23:58:44.885 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/missing 2024-03-17T23:58:44.887 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/mkinstalldirs 2024-03-17T23:58:44.889 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/Makefile.am 2024-03-17T23:58:44.891 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/Makefile.in 2024-03-17T23:58:44.894 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/config.h.in 2024-03-17T23:58:44.896 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/AUTHORS 2024-03-17T23:58:44.898 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/INSTALL 2024-03-17T23:58:44.900 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/ChangeLog 2024-03-17T23:58:44.902 INFO:tasks.workunit.client.0.smithi012.stdout:blogbench-1.0/COPYING 2024-03-17T23:58:44.906 INFO:tasks.workunit.client.0.smithi012.stderr:+ cd blogbench-1.0/ 2024-03-17T23:58:44.906 INFO:tasks.workunit.client.0.smithi012.stdout:making blogbench 2024-03-17T23:58:44.906 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'making blogbench' 2024-03-17T23:58:44.907 INFO:tasks.workunit.client.0.smithi012.stderr:+ ./configure 2024-03-17T23:58:45.223 INFO:tasks.workunit.client.0.smithi012.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-17T23:58:45.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:45 smithi079 ceph-mon[36241]: pgmap v439: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-17T23:58:45.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:45 smithi191 ceph-mon[36708]: pgmap v439: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-17T23:58:45.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:45 smithi012 ceph-mon[31541]: pgmap v439: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-17T23:58:46.245 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether build environment is sane... yes 2024-03-17T23:58:46.269 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gawk... gawk 2024-03-17T23:58:46.300 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether make sets $(MAKE)... yes 2024-03-17T23:58:46.325 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gcc... gcc 2024-03-17T23:58:46.438 INFO:tasks.workunit.client.0.smithi012.stdout:checking for C compiler default output file name... a.out 2024-03-17T23:58:46.446 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether the C compiler works... yes 2024-03-17T23:58:46.450 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether we are cross compiling... no 2024-03-17T23:58:46.532 INFO:tasks.workunit.client.0.smithi012.stdout:checking for suffix of executables... 2024-03-17T23:58:46.612 INFO:tasks.workunit.client.0.smithi012.stdout:checking for suffix of object files... o 2024-03-17T23:58:46.704 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether we are using the GNU C compiler... yes 2024-03-17T23:58:46.797 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether gcc accepts -g... yes 2024-03-17T23:58:46.939 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gcc option to accept ANSI C... none needed 2024-03-17T23:58:47.037 INFO:tasks.workunit.client.0.smithi012.stdout:checking for style of include used by make... GNU 2024-03-17T23:58:47.230 INFO:tasks.workunit.client.0.smithi012.stdout:checking dependency style of gcc... gcc3 2024-03-17T23:58:47.230 INFO:tasks.workunit.client.0.smithi012.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-17T23:58:47.368 INFO:tasks.workunit.client.0.smithi012.stdout:checking how to run the C preprocessor... gcc -E 2024-03-17T23:58:47.479 INFO:tasks.workunit.client.0.smithi012.stdout:checking for egrep... grep -E 2024-03-17T23:58:47.505 INFO:tasks.workunit.client.0.smithi012.stdout:checking for AIX... no 2024-03-17T23:58:47.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:47 smithi079 ceph-mon[36241]: pgmap v440: 129 pgs: 129 active+clean; 795 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-03-17T23:58:47.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:47 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:58:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:47 smithi191 ceph-mon[36708]: pgmap v440: 129 pgs: 129 active+clean; 795 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-03-17T23:58:47.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:47 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:58:47.646 INFO:tasks.workunit.client.0.smithi012.stdout:checking for library containing strerror... none required 2024-03-17T23:58:47.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:47 smithi012 ceph-mon[31541]: pgmap v440: 129 pgs: 129 active+clean; 795 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-03-17T23:58:47.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:47 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:58:47.980 INFO:tasks.workunit.client.0.smithi012.stdout:checking for ANSI C header files... yes 2024-03-17T23:58:48.101 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-17T23:58:48.228 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/types.h... yes 2024-03-17T23:58:48.352 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/stat.h... yes 2024-03-17T23:58:48.465 INFO:tasks.workunit.client.0.smithi012.stdout:checking for stdlib.h... yes 2024-03-17T23:58:48.582 INFO:tasks.workunit.client.0.smithi012.stdout:checking for string.h... yes 2024-03-17T23:58:48.708 INFO:tasks.workunit.client.0.smithi012.stdout:checking for memory.h... yes 2024-03-17T23:58:48.842 INFO:tasks.workunit.client.0.smithi012.stdout:checking for strings.h... yes 2024-03-17T23:58:48.971 INFO:tasks.workunit.client.0.smithi012.stdout:checking for inttypes.h... yes 2024-03-17T23:58:49.102 INFO:tasks.workunit.client.0.smithi012.stdout:checking for stdint.h... yes 2024-03-17T23:58:49.243 INFO:tasks.workunit.client.0.smithi012.stdout:checking for unistd.h... yes 2024-03-17T23:58:49.278 INFO:tasks.workunit.client.0.smithi012.stdout:checking for memory.h... (cached) yes 2024-03-17T23:58:49.300 INFO:tasks.workunit.client.0.smithi012.stdout:checking for strings.h... (cached) yes 2024-03-17T23:58:49.319 INFO:tasks.workunit.client.0.smithi012.stdout:checking for unistd.h... (cached) yes 2024-03-17T23:58:49.451 INFO:tasks.workunit.client.0.smithi012.stdout:checking locale.h usability... yes 2024-03-17T23:58:49.505 INFO:tasks.workunit.client.0.smithi012.stdout:checking locale.h presence... yes 2024-03-17T23:58:49.508 INFO:tasks.workunit.client.0.smithi012.stdout:checking for locale.h... yes 2024-03-17T23:58:49.539 INFO:tasks.workunit.client.0.smithi012.stdout:checking for stdlib.h... (cached) yes 2024-03-17T23:58:49.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:49 smithi079 ceph-mon[36241]: pgmap v441: 129 pgs: 129 active+clean; 795 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-03-17T23:58:49.557 INFO:tasks.workunit.client.0.smithi012.stdout:checking for string.h... (cached) yes 2024-03-17T23:58:49.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:49 smithi191 ceph-mon[36708]: pgmap v441: 129 pgs: 129 active+clean; 795 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-03-17T23:58:49.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:49 smithi012 ceph-mon[31541]: pgmap v441: 129 pgs: 129 active+clean; 795 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-03-17T23:58:49.738 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/socket.h usability... yes 2024-03-17T23:58:49.780 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/socket.h presence... yes 2024-03-17T23:58:49.782 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/socket.h... yes 2024-03-17T23:58:49.894 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/time.h usability... yes 2024-03-17T23:58:49.943 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/time.h presence... yes 2024-03-17T23:58:49.945 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/time.h... yes 2024-03-17T23:58:50.092 INFO:tasks.workunit.client.0.smithi012.stdout:checking fcntl.h usability... yes 2024-03-17T23:58:50.152 INFO:tasks.workunit.client.0.smithi012.stdout:checking fcntl.h presence... yes 2024-03-17T23:58:50.156 INFO:tasks.workunit.client.0.smithi012.stdout:checking for fcntl.h... yes 2024-03-17T23:58:50.314 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/fcntl.h usability... yes 2024-03-17T23:58:50.369 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/fcntl.h presence... yes 2024-03-17T23:58:50.372 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/fcntl.h... yes 2024-03-17T23:58:50.537 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/param.h usability... yes 2024-03-17T23:58:50.601 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/param.h presence... yes 2024-03-17T23:58:50.606 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/param.h... yes 2024-03-17T23:58:50.724 INFO:tasks.workunit.client.0.smithi012.stdout:checking ioctl.h usability... no 2024-03-17T23:58:50.773 INFO:tasks.workunit.client.0.smithi012.stdout:checking ioctl.h presence... no 2024-03-17T23:58:50.777 INFO:tasks.workunit.client.0.smithi012.stdout:checking for ioctl.h... no 2024-03-17T23:58:50.895 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/ioctl.h usability... yes 2024-03-17T23:58:50.948 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/ioctl.h presence... yes 2024-03-17T23:58:50.951 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/ioctl.h... yes 2024-03-17T23:58:51.067 INFO:tasks.workunit.client.0.smithi012.stdout:checking return type of signal handlers... void 2024-03-17T23:58:51.181 INFO:tasks.workunit.client.0.smithi012.stdout:checking for size_t... yes 2024-03-17T23:58:51.292 INFO:tasks.workunit.client.0.smithi012.stdout:checking for dev_t... yes 2024-03-17T23:58:51.414 INFO:tasks.workunit.client.0.smithi012.stdout:checking for short... yes 2024-03-17T23:58:51.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:51 smithi079 ceph-mon[36241]: pgmap v442: 129 pgs: 129 active+clean; 833 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-03-17T23:58:51.582 INFO:tasks.workunit.client.0.smithi012.stdout:checking size of short... 2 2024-03-17T23:58:51.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:51 smithi191 ceph-mon[36708]: pgmap v442: 129 pgs: 129 active+clean; 833 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-03-17T23:58:51.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:51 smithi012 ceph-mon[31541]: pgmap v442: 129 pgs: 129 active+clean; 833 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-03-17T23:58:51.697 INFO:tasks.workunit.client.0.smithi012.stdout:checking for int... yes 2024-03-17T23:58:51.850 INFO:tasks.workunit.client.0.smithi012.stdout:checking size of int... 4 2024-03-17T23:58:51.961 INFO:tasks.workunit.client.0.smithi012.stdout:checking for long... yes 2024-03-17T23:58:52.111 INFO:tasks.workunit.client.0.smithi012.stdout:checking size of long... 8 2024-03-17T23:58:52.222 INFO:tasks.workunit.client.0.smithi012.stdout:checking for long long... yes 2024-03-17T23:58:52.377 INFO:tasks.workunit.client.0.smithi012.stdout:checking size of long long... 8 2024-03-17T23:58:52.522 INFO:tasks.workunit.client.0.smithi012.stdout:checking for ssize_t... yes 2024-03-17T23:58:52.628 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sig_atomic_t... yes 2024-03-17T23:58:52.755 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether gcc needs -traditional... no 2024-03-17T23:58:52.845 INFO:tasks.workunit.client.0.smithi012.stdout:checking for an ANSI C-conforming const... yes 2024-03-17T23:58:52.920 INFO:tasks.workunit.client.0.smithi012.stdout:checking for inline... inline 2024-03-17T23:58:53.050 INFO:tasks.workunit.client.0.smithi012.stdout:checking for working alloca.h... yes 2024-03-17T23:58:53.173 INFO:tasks.workunit.client.0.smithi012.stdout:checking for alloca... yes 2024-03-17T23:58:53.324 INFO:tasks.workunit.client.0.smithi012.stdout:checking for working memcmp... yes 2024-03-17T23:58:53.457 INFO:tasks.workunit.client.0.smithi012.stdout:checking for strftime... yes 2024-03-17T23:58:53.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:53 smithi079 ceph-mon[36241]: pgmap v443: 129 pgs: 129 active+clean; 1.9 MiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 113 KiB/s wr, 16 op/s 2024-03-17T23:58:53.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:53 smithi191 ceph-mon[36708]: pgmap v443: 129 pgs: 129 active+clean; 1.9 MiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 113 KiB/s wr, 16 op/s 2024-03-17T23:58:53.621 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether lstat dereferences a symlink specified with a trailing slash... yes 2024-03-17T23:58:53.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:53 smithi012 ceph-mon[31541]: pgmap v443: 129 pgs: 129 active+clean; 1.9 MiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 113 KiB/s wr, 16 op/s 2024-03-17T23:58:53.772 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether stat accepts an empty string... no 2024-03-17T23:58:53.907 INFO:tasks.workunit.client.0.smithi012.stdout:checking for vprintf... yes 2024-03-17T23:58:54.028 INFO:tasks.workunit.client.0.smithi012.stdout:checking for _doprnt... no 2024-03-17T23:58:54.094 INFO:tasks.workunit.client.0.smithi012.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-17T23:58:54.127 INFO:tasks.workunit.client.0.smithi012.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-17T23:58:54.270 INFO:tasks.workunit.client.0.smithi012.stdout:checking for the pthreads library -lpthreads... no 2024-03-17T23:58:54.395 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether pthreads work without any flags... yes 2024-03-17T23:58:54.491 INFO:tasks.workunit.client.0.smithi012.stdout:checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE 2024-03-17T23:58:54.491 INFO:tasks.workunit.client.0.smithi012.stdout:checking if more special flags are required for pthreads... no 2024-03-17T23:58:54.491 INFO:tasks.workunit.client.0.smithi012.stdout:checking for cc_r... gcc 2024-03-17T23:58:54.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:54 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:58:54.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:54 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:58:54.652 INFO:tasks.workunit.client.0.smithi012.stdout:checking for setrlimit... yes 2024-03-17T23:58:54.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:54 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:58:54.790 INFO:tasks.workunit.client.0.smithi012.stdout:checking for setproctitle... no 2024-03-17T23:58:54.940 INFO:tasks.workunit.client.0.smithi012.stdout:checking for getopt_long... yes 2024-03-17T23:58:55.065 INFO:tasks.workunit.client.0.smithi012.stdout:checking for strtoull... yes 2024-03-17T23:58:55.204 INFO:tasks.workunit.client.0.smithi012.stdout:checking for strtoq... yes 2024-03-17T23:58:55.350 INFO:tasks.workunit.client.0.smithi012.stdout:checking for snprintf... yes 2024-03-17T23:58:55.498 INFO:tasks.workunit.client.0.smithi012.stdout:checking for vsnprintf... yes 2024-03-17T23:58:55.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:55 smithi079 ceph-mon[36241]: pgmap v444: 129 pgs: 129 active+clean; 2.5 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 KiB/s wr, 23 op/s 2024-03-17T23:58:55.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:58:55.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:58:55.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:58:55.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:55 smithi191 ceph-mon[36708]: pgmap v444: 129 pgs: 129 active+clean; 2.5 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 KiB/s wr, 23 op/s 2024-03-17T23:58:55.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:58:55.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:58:55.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:58:55.631 INFO:tasks.workunit.client.0.smithi012.stdout:checking for vfprintf... yes 2024-03-17T23:58:55.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:55 smithi012 ceph-mon[31541]: pgmap v444: 129 pgs: 129 active+clean; 2.5 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 KiB/s wr, 23 op/s 2024-03-17T23:58:55.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-17T23:58:55.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-17T23:58:55.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-17T23:58:55.774 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gethostname... yes 2024-03-17T23:58:55.939 INFO:tasks.workunit.client.0.smithi012.stdout:checking for setlocale... yes 2024-03-17T23:58:56.101 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sync... yes 2024-03-17T23:58:56.253 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether snprintf is C99 conformant... done 2024-03-17T23:58:56.284 INFO:tasks.workunit.client.0.smithi012.stdout:configure: creating ./config.status 2024-03-17T23:58:56.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:56 smithi191 ceph-mon[36708]: mgrmap e20: x(active, since 12m), standbys: z, y 2024-03-17T23:58:56.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:56 smithi012 ceph-mon[31541]: mgrmap e20: x(active, since 12m), standbys: z, y 2024-03-17T23:58:56.680 INFO:tasks.workunit.client.0.smithi012.stdout:config.status: creating Makefile 2024-03-17T23:58:56.747 INFO:tasks.workunit.client.0.smithi012.stdout:config.status: creating src/Makefile 2024-03-17T23:58:56.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:56 smithi079 ceph-mon[36241]: mgrmap e20: x(active, since 12m), standbys: z, y 2024-03-17T23:58:57.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:57 smithi191 ceph-mon[36708]: pgmap v445: 129 pgs: 129 active+clean; 3.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 291 KiB/s wr, 36 op/s 2024-03-17T23:58:57.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:57 smithi012 ceph-mon[31541]: pgmap v445: 129 pgs: 129 active+clean; 3.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 291 KiB/s wr, 36 op/s 2024-03-17T23:58:57.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:57 smithi079 ceph-mon[36241]: pgmap v445: 129 pgs: 129 active+clean; 3.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 291 KiB/s wr, 36 op/s 2024-03-17T23:58:59.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:58:59 smithi191 ceph-mon[36708]: pgmap v446: 129 pgs: 129 active+clean; 3.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 302 KiB/s wr, 39 op/s 2024-03-17T23:58:59.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:58:59 smithi012 ceph-mon[31541]: pgmap v446: 129 pgs: 129 active+clean; 3.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 302 KiB/s wr, 39 op/s 2024-03-17T23:58:59.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:58:59 smithi079 ceph-mon[36241]: pgmap v446: 129 pgs: 129 active+clean; 3.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 302 KiB/s wr, 39 op/s 2024-03-17T23:59:00.365 INFO:tasks.workunit.client.0.smithi012.stdout:config.status: creating man/Makefile 2024-03-17T23:59:00.396 INFO:tasks.workunit.client.0.smithi012.stdout:config.status: creating config.h 2024-03-17T23:59:00.534 INFO:tasks.workunit.client.0.smithi012.stdout:config.status: executing depfiles commands 2024-03-17T23:59:00.901 INFO:tasks.workunit.client.0.smithi012.stderr:+ make 2024-03-17T23:59:00.920 INFO:tasks.workunit.client.0.smithi012.stdout:make all-recursive 2024-03-17T23:59:00.925 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-17T23:59:00.934 INFO:tasks.workunit.client.0.smithi012.stdout:Making all in src 2024-03-17T23:59:00.952 INFO:tasks.workunit.client.0.smithi012.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-17T23:59:00.953 INFO:tasks.workunit.client.0.smithi012.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT blogbench.o -MD -MP -MF ".deps/blogbench.Tpo" -c -o blogbench.o blogbench.c; \ 2024-03-17T23:59:00.953 INFO:tasks.workunit.client.0.smithi012.stdout:then mv -f ".deps/blogbench.Tpo" ".deps/blogbench.Po"; else rm -f ".deps/blogbench.Tpo"; exit 1; fi 2024-03-17T23:59:01.152 INFO:tasks.workunit.client.0.smithi012.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT process.o -MD -MP -MF ".deps/process.Tpo" -c -o process.o process.c; \ 2024-03-17T23:59:01.152 INFO:tasks.workunit.client.0.smithi012.stdout:then mv -f ".deps/process.Tpo" ".deps/process.Po"; else rm -f ".deps/process.Tpo"; exit 1; fi 2024-03-17T23:59:01.327 INFO:tasks.workunit.client.0.smithi012.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT writer.o -MD -MP -MF ".deps/writer.Tpo" -c -o writer.o writer.c; \ 2024-03-17T23:59:01.327 INFO:tasks.workunit.client.0.smithi012.stdout:then mv -f ".deps/writer.Tpo" ".deps/writer.Po"; else rm -f ".deps/writer.Tpo"; exit 1; fi 2024-03-17T23:59:01.482 INFO:tasks.workunit.client.0.smithi012.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT rewriter.o -MD -MP -MF ".deps/rewriter.Tpo" -c -o rewriter.o rewriter.c; \ 2024-03-17T23:59:01.482 INFO:tasks.workunit.client.0.smithi012.stdout:then mv -f ".deps/rewriter.Tpo" ".deps/rewriter.Po"; else rm -f ".deps/rewriter.Tpo"; exit 1; fi 2024-03-17T23:59:01.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:01 smithi191 ceph-mon[36708]: pgmap v447: 129 pgs: 129 active+clean; 3.7 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 326 KiB/s wr, 51 op/s 2024-03-17T23:59:01.647 INFO:tasks.workunit.client.0.smithi012.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT reader.o -MD -MP -MF ".deps/reader.Tpo" -c -o reader.o reader.c; \ 2024-03-17T23:59:01.647 INFO:tasks.workunit.client.0.smithi012.stdout:then mv -f ".deps/reader.Tpo" ".deps/reader.Po"; else rm -f ".deps/reader.Tpo"; exit 1; fi 2024-03-17T23:59:01.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:01 smithi012 ceph-mon[31541]: pgmap v447: 129 pgs: 129 active+clean; 3.7 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 326 KiB/s wr, 51 op/s 2024-03-17T23:59:01.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:01 smithi079 ceph-mon[36241]: pgmap v447: 129 pgs: 129 active+clean; 3.7 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 326 KiB/s wr, 51 op/s 2024-03-17T23:59:01.823 INFO:tasks.workunit.client.0.smithi012.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT commenter.o -MD -MP -MF ".deps/commenter.Tpo" -c -o commenter.o commenter.c; \ 2024-03-17T23:59:01.824 INFO:tasks.workunit.client.0.smithi012.stdout:then mv -f ".deps/commenter.Tpo" ".deps/commenter.Po"; else rm -f ".deps/commenter.Tpo"; exit 1; fi 2024-03-17T23:59:01.986 INFO:tasks.workunit.client.0.smithi012.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT helpers.o -MD -MP -MF ".deps/helpers.Tpo" -c -o helpers.o helpers.c; \ 2024-03-17T23:59:01.986 INFO:tasks.workunit.client.0.smithi012.stdout:then mv -f ".deps/helpers.Tpo" ".deps/helpers.Po"; else rm -f ".deps/helpers.Tpo"; exit 1; fi 2024-03-17T23:59:02.177 INFO:tasks.workunit.client.0.smithi012.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT bsd-getopt_long.o -MD -MP -MF ".deps/bsd-getopt_long.Tpo" -c -o bsd-getopt_long.o bsd-getopt_long.c; \ 2024-03-17T23:59:02.177 INFO:tasks.workunit.client.0.smithi012.stdout:then mv -f ".deps/bsd-getopt_long.Tpo" ".deps/bsd-getopt_long.Po"; else rm -f ".deps/bsd-getopt_long.Tpo"; exit 1; fi 2024-03-17T23:59:02.315 INFO:tasks.workunit.client.0.smithi012.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT fakesnprintf.o -MD -MP -MF ".deps/fakesnprintf.Tpo" -c -o fakesnprintf.o fakesnprintf.c; \ 2024-03-17T23:59:02.316 INFO:tasks.workunit.client.0.smithi012.stdout:then mv -f ".deps/fakesnprintf.Tpo" ".deps/fakesnprintf.Po"; else rm -f ".deps/fakesnprintf.Tpo"; exit 1; fi 2024-03-17T23:59:02.467 INFO:tasks.workunit.client.0.smithi012.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT mysnprintf.o -MD -MP -MF ".deps/mysnprintf.Tpo" -c -o mysnprintf.o mysnprintf.c; \ 2024-03-17T23:59:02.467 INFO:tasks.workunit.client.0.smithi012.stdout:then mv -f ".deps/mysnprintf.Tpo" ".deps/mysnprintf.Po"; else rm -f ".deps/mysnprintf.Tpo"; exit 1; fi 2024-03-17T23:59:02.593 INFO:tasks.workunit.client.0.smithi012.stdout:gcc -g -O2 -o blogbench blogbench.o process.o writer.o rewriter.o reader.o commenter.o helpers.o bsd-getopt_long.o fakesnprintf.o mysnprintf.o 2024-03-17T23:59:02.644 INFO:tasks.workunit.client.0.smithi012.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-17T23:59:02.644 INFO:tasks.workunit.client.0.smithi012.stdout:Making all in man 2024-03-17T23:59:02.654 INFO:tasks.workunit.client.0.smithi012.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-17T23:59:02.654 INFO:tasks.workunit.client.0.smithi012.stdout:make[2]: Nothing to be done for 'all'. 2024-03-17T23:59:02.654 INFO:tasks.workunit.client.0.smithi012.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-17T23:59:02.659 INFO:tasks.workunit.client.0.smithi012.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-17T23:59:02.663 INFO:tasks.workunit.client.0.smithi012.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-17T23:59:02.663 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-17T23:59:02.665 INFO:tasks.workunit.client.0.smithi012.stderr:+ cd src 2024-03-17T23:59:02.672 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir blogtest_in 2024-03-17T23:59:02.678 INFO:tasks.workunit.client.0.smithi012.stdout:running blogbench 2024-03-17T23:59:02.678 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'running blogbench' 2024-03-17T23:59:02.678 INFO:tasks.workunit.client.0.smithi012.stderr:+ ./blogbench -d blogtest_in 2024-03-17T23:59:02.680 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-03-17T23:59:02.680 INFO:tasks.workunit.client.0.smithi012.stdout:Frequency = 10 secs 2024-03-17T23:59:02.680 INFO:tasks.workunit.client.0.smithi012.stdout:Scratch dir = [blogtest_in] 2024-03-17T23:59:03.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:03 smithi191 ceph-mon[36708]: pgmap v448: 129 pgs: 129 active+clean; 4.1 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 356 KiB/s wr, 53 op/s 2024-03-17T23:59:03.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:03 smithi012 ceph-mon[31541]: pgmap v448: 129 pgs: 129 active+clean; 4.1 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 356 KiB/s wr, 53 op/s 2024-03-17T23:59:03.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:03 smithi079 ceph-mon[36241]: pgmap v448: 129 pgs: 129 active+clean; 4.1 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 356 KiB/s wr, 53 op/s 2024-03-17T23:59:04.489 INFO:tasks.workunit.client.0.smithi012.stdout:Spawning 3 writers... 2024-03-17T23:59:04.490 INFO:tasks.workunit.client.0.smithi012.stdout:Spawning 1 rewriters... 2024-03-17T23:59:04.490 INFO:tasks.workunit.client.0.smithi012.stdout:Spawning 5 commenters... 2024-03-17T23:59:04.490 INFO:tasks.workunit.client.0.smithi012.stdout:Spawning 100 readers... 2024-03-17T23:59:04.498 INFO:tasks.workunit.client.0.smithi012.stdout:Benchmarking for 30 iterations. 2024-03-17T23:59:04.499 INFO:tasks.workunit.client.0.smithi012.stdout:The test will run during 5 minutes. 2024-03-17T23:59:04.499 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-03-17T23:59:04.499 INFO:tasks.workunit.client.0.smithi012.stdout: Nb blogs R articles W articles R pictures W pictures R comments W comments 2024-03-17T23:59:05.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:05 smithi191 ceph-mon[36708]: pgmap v449: 129 pgs: 129 active+clean; 4.2 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 275 KiB/s wr, 54 op/s 2024-03-17T23:59:05.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:05 smithi012 ceph-mon[31541]: pgmap v449: 129 pgs: 129 active+clean; 4.2 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 275 KiB/s wr, 54 op/s 2024-03-17T23:59:05.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:05 smithi079 ceph-mon[36241]: pgmap v449: 129 pgs: 129 active+clean; 4.2 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 275 KiB/s wr, 54 op/s 2024-03-17T23:59:07.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:07 smithi191 ceph-mon[36708]: pgmap v450: 129 pgs: 129 active+clean; 4.6 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 275 KiB/s wr, 59 op/s 2024-03-17T23:59:07.726 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:07 smithi012 ceph-mon[31541]: pgmap v450: 129 pgs: 129 active+clean; 4.6 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 275 KiB/s wr, 59 op/s 2024-03-17T23:59:07.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:07 smithi079 ceph-mon[36241]: pgmap v450: 129 pgs: 129 active+clean; 4.6 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 275 KiB/s wr, 59 op/s 2024-03-17T23:59:09.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:09 smithi012 ceph-mon[31541]: pgmap v451: 129 pgs: 129 active+clean; 4.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 164 KiB/s wr, 46 op/s 2024-03-17T23:59:09.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:09 smithi079 ceph-mon[36241]: pgmap v451: 129 pgs: 129 active+clean; 4.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 164 KiB/s wr, 46 op/s 2024-03-17T23:59:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:09 smithi191 ceph-mon[36708]: pgmap v451: 129 pgs: 129 active+clean; 4.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 164 KiB/s wr, 46 op/s 2024-03-17T23:59:11.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:11 smithi012 ceph-mon[31541]: pgmap v452: 129 pgs: 129 active+clean; 8.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 496 KiB/s wr, 47 op/s 2024-03-17T23:59:11.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:11 smithi079 ceph-mon[36241]: pgmap v452: 129 pgs: 129 active+clean; 8.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 496 KiB/s wr, 47 op/s 2024-03-17T23:59:11.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:11 smithi191 ceph-mon[36708]: pgmap v452: 129 pgs: 129 active+clean; 8.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 496 KiB/s wr, 47 op/s 2024-03-17T23:59:12.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:12 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:59:12.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:12 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:59:12.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:12 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:59:13.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:13 smithi012 ceph-mon[31541]: pgmap v453: 129 pgs: 129 active+clean; 15 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1022 KiB/s wr, 51 op/s 2024-03-17T23:59:13.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:13 smithi079 ceph-mon[36241]: pgmap v453: 129 pgs: 129 active+clean; 15 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1022 KiB/s wr, 51 op/s 2024-03-17T23:59:13.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:13 smithi191 ceph-mon[36708]: pgmap v453: 129 pgs: 129 active+clean; 15 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1022 KiB/s wr, 51 op/s 2024-03-17T23:59:14.499 INFO:tasks.workunit.client.0.smithi012.stdout: 16 181672 893 143170 760 37888 763 2024-03-17T23:59:15.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:15 smithi012 ceph-mon[31541]: pgmap v454: 129 pgs: 129 active+clean; 31 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 111 op/s 2024-03-17T23:59:15.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:15 smithi079 ceph-mon[36241]: pgmap v454: 129 pgs: 129 active+clean; 31 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 111 op/s 2024-03-17T23:59:15.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:15 smithi191 ceph-mon[36708]: pgmap v454: 129 pgs: 129 active+clean; 31 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 111 op/s 2024-03-17T23:59:17.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:17 smithi012 ceph-mon[31541]: pgmap v455: 129 pgs: 129 active+clean; 47 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 178 op/s 2024-03-17T23:59:17.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:17 smithi079 ceph-mon[36241]: pgmap v455: 129 pgs: 129 active+clean; 47 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 178 op/s 2024-03-17T23:59:17.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:17 smithi191 ceph-mon[36708]: pgmap v455: 129 pgs: 129 active+clean; 47 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 178 op/s 2024-03-17T23:59:19.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:19 smithi012 ceph-mon[31541]: pgmap v456: 129 pgs: 129 active+clean; 47 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 171 op/s 2024-03-17T23:59:19.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:19 smithi079 ceph-mon[36241]: pgmap v456: 129 pgs: 129 active+clean; 47 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 171 op/s 2024-03-17T23:59:19.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:19 smithi191 ceph-mon[36708]: pgmap v456: 129 pgs: 129 active+clean; 47 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 171 op/s 2024-03-17T23:59:21.675 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:21 smithi012 ceph-mon[31541]: pgmap v457: 129 pgs: 129 active+clean; 48 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 178 op/s 2024-03-17T23:59:21.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:21 smithi079 ceph-mon[36241]: pgmap v457: 129 pgs: 129 active+clean; 48 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 178 op/s 2024-03-17T23:59:21.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:21 smithi191 ceph-mon[36708]: pgmap v457: 129 pgs: 129 active+clean; 48 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 178 op/s 2024-03-17T23:59:23.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:23 smithi012 ceph-mon[31541]: pgmap v458: 129 pgs: 129 active+clean; 52 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 182 op/s 2024-03-17T23:59:23.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:23 smithi079 ceph-mon[36241]: pgmap v458: 129 pgs: 129 active+clean; 52 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 182 op/s 2024-03-17T23:59:23.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:23 smithi191 ceph-mon[36708]: pgmap v458: 129 pgs: 129 active+clean; 52 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 182 op/s 2024-03-17T23:59:24.499 INFO:tasks.workunit.client.0.smithi012.stdout: 31 218686 770 158646 770 59543 719 2024-03-17T23:59:25.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:25 smithi012 ceph-mon[31541]: pgmap v459: 129 pgs: 129 active+clean; 52 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 175 op/s 2024-03-17T23:59:25.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:25 smithi079 ceph-mon[36241]: pgmap v459: 129 pgs: 129 active+clean; 52 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 175 op/s 2024-03-17T23:59:25.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:25 smithi191 ceph-mon[36708]: pgmap v459: 129 pgs: 129 active+clean; 52 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 175 op/s 2024-03-17T23:59:27.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:27 smithi012 ceph-mon[31541]: pgmap v460: 129 pgs: 129 active+clean; 58 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 125 op/s 2024-03-17T23:59:27.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:27 smithi079 ceph-mon[36241]: pgmap v460: 129 pgs: 129 active+clean; 58 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 125 op/s 2024-03-17T23:59:27.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:27 smithi191 ceph-mon[36708]: pgmap v460: 129 pgs: 129 active+clean; 58 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 125 op/s 2024-03-17T23:59:29.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:29 smithi012 ceph-mon[31541]: pgmap v461: 129 pgs: 129 active+clean; 65 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-03-17T23:59:29.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:29 smithi079 ceph-mon[36241]: pgmap v461: 129 pgs: 129 active+clean; 65 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-03-17T23:59:29.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:29 smithi191 ceph-mon[36708]: pgmap v461: 129 pgs: 129 active+clean; 65 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-03-17T23:59:31.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:31 smithi079 ceph-mon[36241]: pgmap v462: 129 pgs: 129 active+clean; 80 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 141 op/s 2024-03-17T23:59:31.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:31 smithi191 ceph-mon[36708]: pgmap v462: 129 pgs: 129 active+clean; 80 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 141 op/s 2024-03-17T23:59:31.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:31 smithi012 ceph-mon[31541]: pgmap v462: 129 pgs: 129 active+clean; 80 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 141 op/s 2024-03-17T23:59:33.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:33 smithi079 ceph-mon[36241]: pgmap v463: 129 pgs: 129 active+clean; 84 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 162 op/s 2024-03-17T23:59:33.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:33 smithi191 ceph-mon[36708]: pgmap v463: 129 pgs: 129 active+clean; 84 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 162 op/s 2024-03-17T23:59:33.929 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:33 smithi012 ceph-mon[31541]: pgmap v463: 129 pgs: 129 active+clean; 84 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 162 op/s 2024-03-17T23:59:34.512 INFO:tasks.workunit.client.0.smithi012.stdout: 34 480899 255 340003 271 183070 810 2024-03-17T23:59:35.728 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:35 smithi012 ceph-mon[31541]: pgmap v464: 129 pgs: 129 active+clean; 108 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.1 MiB/s wr, 247 op/s 2024-03-17T23:59:35.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:35 smithi079 ceph-mon[36241]: pgmap v464: 129 pgs: 129 active+clean; 108 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.1 MiB/s wr, 247 op/s 2024-03-17T23:59:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:35 smithi191 ceph-mon[36708]: pgmap v464: 129 pgs: 129 active+clean; 108 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.1 MiB/s wr, 247 op/s 2024-03-17T23:59:37.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:37 smithi079 ceph-mon[36241]: pgmap v465: 129 pgs: 129 active+clean; 129 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.0 MiB/s wr, 347 op/s 2024-03-17T23:59:37.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:37 smithi191 ceph-mon[36708]: pgmap v465: 129 pgs: 129 active+clean; 129 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.0 MiB/s wr, 347 op/s 2024-03-17T23:59:37.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:37 smithi012 ceph-mon[31541]: pgmap v465: 129 pgs: 129 active+clean; 129 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.0 MiB/s wr, 347 op/s 2024-03-17T23:59:39.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:39 smithi079 ceph-mon[36241]: pgmap v466: 129 pgs: 129 active+clean; 136 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.1 MiB/s wr, 356 op/s 2024-03-17T23:59:39.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:39 smithi191 ceph-mon[36708]: pgmap v466: 129 pgs: 129 active+clean; 136 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.1 MiB/s wr, 356 op/s 2024-03-17T23:59:39.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:39 smithi012 ceph-mon[31541]: pgmap v466: 129 pgs: 129 active+clean; 136 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.1 MiB/s wr, 356 op/s 2024-03-17T23:59:41.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:41 smithi079 ceph-mon[36241]: pgmap v467: 129 pgs: 129 active+clean; 137 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.7 MiB/s wr, 346 op/s 2024-03-17T23:59:41.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:41 smithi191 ceph-mon[36708]: pgmap v467: 129 pgs: 129 active+clean; 137 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.7 MiB/s wr, 346 op/s 2024-03-17T23:59:41.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:41 smithi012 ceph-mon[31541]: pgmap v467: 129 pgs: 129 active+clean; 137 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.7 MiB/s wr, 346 op/s 2024-03-17T23:59:43.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:43 smithi079 ceph-mon[36241]: pgmap v468: 129 pgs: 129 active+clean; 137 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.4 MiB/s wr, 289 op/s 2024-03-17T23:59:43.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:43 smithi191 ceph-mon[36708]: pgmap v468: 129 pgs: 129 active+clean; 137 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.4 MiB/s wr, 289 op/s 2024-03-17T23:59:43.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:43 smithi012 ceph-mon[31541]: pgmap v468: 129 pgs: 129 active+clean; 137 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.4 MiB/s wr, 289 op/s 2024-03-17T23:59:44.512 INFO:tasks.workunit.client.0.smithi012.stdout: 42 547614 462 407623 425 252613 1298 2024-03-17T23:59:45.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:45 smithi079 ceph-mon[36241]: pgmap v469: 129 pgs: 129 active+clean; 142 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 274 op/s 2024-03-17T23:59:45.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:45 smithi191 ceph-mon[36708]: pgmap v469: 129 pgs: 129 active+clean; 142 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 274 op/s 2024-03-17T23:59:45.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:45 smithi012 ceph-mon[31541]: pgmap v469: 129 pgs: 129 active+clean; 142 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 274 op/s 2024-03-17T23:59:47.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:47 smithi079 ceph-mon[36241]: pgmap v470: 129 pgs: 129 active+clean; 145 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.7 MiB/s wr, 198 op/s 2024-03-17T23:59:47.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:47 smithi191 ceph-mon[36708]: pgmap v470: 129 pgs: 129 active+clean; 145 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.7 MiB/s wr, 198 op/s 2024-03-17T23:59:47.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:47 smithi012 ceph-mon[31541]: pgmap v470: 129 pgs: 129 active+clean; 145 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.7 MiB/s wr, 198 op/s 2024-03-17T23:59:49.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:49 smithi079 ceph-mon[36241]: pgmap v471: 129 pgs: 129 active+clean; 145 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 94 op/s 2024-03-17T23:59:49.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:49 smithi191 ceph-mon[36708]: pgmap v471: 129 pgs: 129 active+clean; 145 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 94 op/s 2024-03-17T23:59:49.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:49 smithi012 ceph-mon[31541]: pgmap v471: 129 pgs: 129 active+clean; 145 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 94 op/s 2024-03-17T23:59:51.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:51 smithi079 ceph-mon[36241]: pgmap v472: 129 pgs: 129 active+clean; 148 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-17T23:59:51.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:51 smithi191 ceph-mon[36708]: pgmap v472: 129 pgs: 129 active+clean; 148 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-17T23:59:51.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:51 smithi012 ceph-mon[31541]: pgmap v472: 129 pgs: 129 active+clean; 148 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-17T23:59:52.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:52 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:59:52.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:52 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:59:52.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:52 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:59:53.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:53 smithi079 ceph-mon[36241]: pgmap v473: 129 pgs: 129 active+clean; 148 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.2 MiB/s wr, 69 op/s 2024-03-17T23:59:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:53 smithi191 ceph-mon[36708]: pgmap v473: 129 pgs: 129 active+clean; 148 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.2 MiB/s wr, 69 op/s 2024-03-17T23:59:53.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:53 smithi012 ceph-mon[31541]: pgmap v473: 129 pgs: 129 active+clean; 148 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.2 MiB/s wr, 69 op/s 2024-03-17T23:59:54.515 INFO:tasks.workunit.client.0.smithi012.stdout: 47 504721 295 361025 337 251555 991 2024-03-17T23:59:55.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:55 smithi079 ceph-mon[36241]: pgmap v474: 129 pgs: 129 active+clean; 161 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-03-17T23:59:55.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:59:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:55 smithi191 ceph-mon[36708]: pgmap v474: 129 pgs: 129 active+clean; 161 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-03-17T23:59:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:59:55.926 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:55 smithi012 ceph-mon[31541]: pgmap v474: 129 pgs: 129 active+clean; 161 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-03-17T23:59:55.926 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-17T23:59:57.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:57 smithi079 ceph-mon[36241]: pgmap v475: 129 pgs: 129 active+clean; 171 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-17T23:59:57.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:57 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:59:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:57 smithi191 ceph-mon[36708]: pgmap v475: 129 pgs: 129 active+clean; 171 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-17T23:59:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:57 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:59:57.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:57 smithi012 ceph-mon[31541]: pgmap v475: 129 pgs: 129 active+clean; 171 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-17T23:59:57.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:57 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-17T23:59:59.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 17 23:59:59 smithi079 ceph-mon[36241]: pgmap v476: 129 pgs: 129 active+clean; 177 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 211 op/s 2024-03-17T23:59:59.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 17 23:59:59 smithi191 ceph-mon[36708]: pgmap v476: 129 pgs: 129 active+clean; 177 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 211 op/s 2024-03-17T23:59:59.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 17 23:59:59 smithi012 ceph-mon[31541]: pgmap v476: 129 pgs: 129 active+clean; 177 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 211 op/s 2024-03-18T00:00:00.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:00 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:00:00.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:00 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:00:00.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:00 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:00:00.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:00 smithi191 ceph-mon[36708]: overall HEALTH_OK 2024-03-18T00:00:00.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:00 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:00:00.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:00 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:00:00.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:00 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:00:00.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:00 smithi012 ceph-mon[31541]: overall HEALTH_OK 2024-03-18T00:00:01.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:00 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:00:01.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:00 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:00:01.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:00 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:00:01.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:00 smithi079 ceph-mon[36241]: overall HEALTH_OK 2024-03-18T00:00:02.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:01 smithi079 ceph-mon[36241]: pgmap v477: 129 pgs: 129 active+clean; 209 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 476 op/s 2024-03-18T00:00:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:01 smithi191 ceph-mon[36708]: pgmap v477: 129 pgs: 129 active+clean; 209 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 476 op/s 2024-03-18T00:00:02.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:01 smithi012 ceph-mon[31541]: pgmap v477: 129 pgs: 129 active+clean; 209 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 476 op/s 2024-03-18T00:00:04.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:03 smithi079 ceph-mon[36241]: pgmap v478: 129 pgs: 129 active+clean; 220 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.7 MiB/s wr, 531 op/s 2024-03-18T00:00:04.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:03 smithi191 ceph-mon[36708]: pgmap v478: 129 pgs: 129 active+clean; 220 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.7 MiB/s wr, 531 op/s 2024-03-18T00:00:04.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:03 smithi012 ceph-mon[31541]: pgmap v478: 129 pgs: 129 active+clean; 220 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.7 MiB/s wr, 531 op/s 2024-03-18T00:00:04.525 INFO:tasks.workunit.client.0.smithi012.stdout: 51 302781 238 221743 215 158775 748 2024-03-18T00:00:06.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:05 smithi079 ceph-mon[36241]: pgmap v479: 129 pgs: 129 active+clean; 225 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.3 MiB/s wr, 596 op/s 2024-03-18T00:00:06.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:05 smithi191 ceph-mon[36708]: pgmap v479: 129 pgs: 129 active+clean; 225 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.3 MiB/s wr, 596 op/s 2024-03-18T00:00:06.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:05 smithi012 ceph-mon[31541]: pgmap v479: 129 pgs: 129 active+clean; 225 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.3 MiB/s wr, 596 op/s 2024-03-18T00:00:08.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:07 smithi079 ceph-mon[36241]: pgmap v480: 129 pgs: 129 active+clean; 227 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.5 MiB/s wr, 601 op/s 2024-03-18T00:00:08.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:07 smithi191 ceph-mon[36708]: pgmap v480: 129 pgs: 129 active+clean; 227 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.5 MiB/s wr, 601 op/s 2024-03-18T00:00:08.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:07 smithi012 ceph-mon[31541]: pgmap v480: 129 pgs: 129 active+clean; 227 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.5 MiB/s wr, 601 op/s 2024-03-18T00:00:10.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:09 smithi079 ceph-mon[36241]: pgmap v481: 129 pgs: 129 active+clean; 230 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.9 MiB/s wr, 539 op/s 2024-03-18T00:00:10.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:09 smithi191 ceph-mon[36708]: pgmap v481: 129 pgs: 129 active+clean; 230 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.9 MiB/s wr, 539 op/s 2024-03-18T00:00:10.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:09 smithi012 ceph-mon[31541]: pgmap v481: 129 pgs: 129 active+clean; 230 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.9 MiB/s wr, 539 op/s 2024-03-18T00:00:12.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:11 smithi079 ceph-mon[36241]: pgmap v482: 129 pgs: 129 active+clean; 232 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.6 MiB/s wr, 533 op/s 2024-03-18T00:00:12.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:11 smithi191 ceph-mon[36708]: pgmap v482: 129 pgs: 129 active+clean; 232 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.6 MiB/s wr, 533 op/s 2024-03-18T00:00:12.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:11 smithi012 ceph-mon[31541]: pgmap v482: 129 pgs: 129 active+clean; 232 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.6 MiB/s wr, 533 op/s 2024-03-18T00:00:14.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:13 smithi079 ceph-mon[36241]: pgmap v483: 129 pgs: 129 active+clean; 235 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 275 op/s 2024-03-18T00:00:14.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:13 smithi191 ceph-mon[36708]: pgmap v483: 129 pgs: 129 active+clean; 235 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 275 op/s 2024-03-18T00:00:14.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:13 smithi012 ceph-mon[31541]: pgmap v483: 129 pgs: 129 active+clean; 235 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 275 op/s 2024-03-18T00:00:14.521 INFO:tasks.workunit.client.0.smithi012.stdout: 58 534914 583 381621 287 276239 1200 2024-03-18T00:00:16.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:15 smithi079 ceph-mon[36241]: pgmap v484: 129 pgs: 129 active+clean; 234 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 220 op/s 2024-03-18T00:00:16.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:15 smithi191 ceph-mon[36708]: pgmap v484: 129 pgs: 129 active+clean; 234 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 220 op/s 2024-03-18T00:00:16.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:15 smithi012 ceph-mon[31541]: pgmap v484: 129 pgs: 129 active+clean; 234 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 220 op/s 2024-03-18T00:00:18.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:17 smithi079 ceph-mon[36241]: pgmap v485: 129 pgs: 129 active+clean; 239 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-03-18T00:00:18.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:17 smithi191 ceph-mon[36708]: pgmap v485: 129 pgs: 129 active+clean; 239 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-03-18T00:00:18.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:17 smithi012 ceph-mon[31541]: pgmap v485: 129 pgs: 129 active+clean; 239 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-03-18T00:00:20.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:19 smithi079 ceph-mon[36241]: pgmap v486: 129 pgs: 129 active+clean; 243 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 150 op/s 2024-03-18T00:00:20.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:19 smithi191 ceph-mon[36708]: pgmap v486: 129 pgs: 129 active+clean; 243 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 150 op/s 2024-03-18T00:00:20.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:19 smithi012 ceph-mon[31541]: pgmap v486: 129 pgs: 129 active+clean; 243 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 150 op/s 2024-03-18T00:00:22.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:21 smithi079 ceph-mon[36241]: pgmap v487: 129 pgs: 129 active+clean; 242 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 220 op/s 2024-03-18T00:00:22.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:21 smithi191 ceph-mon[36708]: pgmap v487: 129 pgs: 129 active+clean; 242 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 220 op/s 2024-03-18T00:00:22.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:21 smithi012 ceph-mon[31541]: pgmap v487: 129 pgs: 129 active+clean; 242 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 220 op/s 2024-03-18T00:00:24.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:23 smithi079 ceph-mon[36241]: pgmap v488: 129 pgs: 129 active+clean; 243 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 195 op/s 2024-03-18T00:00:24.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:23 smithi191 ceph-mon[36708]: pgmap v488: 129 pgs: 129 active+clean; 243 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 195 op/s 2024-03-18T00:00:24.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:23 smithi012 ceph-mon[31541]: pgmap v488: 129 pgs: 129 active+clean; 243 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 195 op/s 2024-03-18T00:00:24.521 INFO:tasks.workunit.client.0.smithi012.stdout: 67 160550 512 111386 375 80924 1095 2024-03-18T00:00:26.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:25 smithi079 ceph-mon[36241]: pgmap v489: 129 pgs: 129 active+clean; 248 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 211 op/s 2024-03-18T00:00:26.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:25 smithi191 ceph-mon[36708]: pgmap v489: 129 pgs: 129 active+clean; 248 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 211 op/s 2024-03-18T00:00:26.220 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:25 smithi012 ceph-mon[31541]: pgmap v489: 129 pgs: 129 active+clean; 248 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 211 op/s 2024-03-18T00:00:28.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:27 smithi191 ceph-mon[36708]: pgmap v490: 129 pgs: 129 active+clean; 255 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 255 op/s 2024-03-18T00:00:28.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:27 smithi012 ceph-mon[31541]: pgmap v490: 129 pgs: 129 active+clean; 255 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 255 op/s 2024-03-18T00:00:28.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:27 smithi079 ceph-mon[36241]: pgmap v490: 129 pgs: 129 active+clean; 255 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 255 op/s 2024-03-18T00:00:30.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:29 smithi191 ceph-mon[36708]: pgmap v491: 129 pgs: 129 active+clean; 260 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 252 op/s 2024-03-18T00:00:30.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:29 smithi012 ceph-mon[31541]: pgmap v491: 129 pgs: 129 active+clean; 260 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 252 op/s 2024-03-18T00:00:30.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:29 smithi079 ceph-mon[36241]: pgmap v491: 129 pgs: 129 active+clean; 260 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 252 op/s 2024-03-18T00:00:32.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:31 smithi191 ceph-mon[36708]: pgmap v492: 129 pgs: 129 active+clean; 281 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 380 op/s 2024-03-18T00:00:32.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:31 smithi012 ceph-mon[31541]: pgmap v492: 129 pgs: 129 active+clean; 281 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 380 op/s 2024-03-18T00:00:32.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:31 smithi079 ceph-mon[36241]: pgmap v492: 129 pgs: 129 active+clean; 281 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.9 MiB/s wr, 380 op/s 2024-03-18T00:00:34.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:33 smithi191 ceph-mon[36708]: pgmap v493: 129 pgs: 129 active+clean; 286 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 330 op/s 2024-03-18T00:00:34.181 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:33 smithi012 ceph-mon[31541]: pgmap v493: 129 pgs: 129 active+clean; 286 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 330 op/s 2024-03-18T00:00:34.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:33 smithi079 ceph-mon[36241]: pgmap v493: 129 pgs: 129 active+clean; 286 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 330 op/s 2024-03-18T00:00:34.555 INFO:tasks.workunit.client.0.smithi012.stdout: 70 337247 246 234642 214 178616 717 2024-03-18T00:00:36.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:35 smithi191 ceph-mon[36708]: pgmap v494: 129 pgs: 129 active+clean; 314 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.8 MiB/s wr, 485 op/s 2024-03-18T00:00:36.177 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:35 smithi012 ceph-mon[31541]: pgmap v494: 129 pgs: 129 active+clean; 314 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.8 MiB/s wr, 485 op/s 2024-03-18T00:00:36.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:35 smithi079 ceph-mon[36241]: pgmap v494: 129 pgs: 129 active+clean; 314 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.8 MiB/s wr, 485 op/s 2024-03-18T00:00:38.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:37 smithi191 ceph-mon[36708]: pgmap v495: 129 pgs: 129 active+clean; 327 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.6 MiB/s wr, 569 op/s 2024-03-18T00:00:38.178 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:37 smithi012 ceph-mon[31541]: pgmap v495: 129 pgs: 129 active+clean; 327 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.6 MiB/s wr, 569 op/s 2024-03-18T00:00:38.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:37 smithi079 ceph-mon[36241]: pgmap v495: 129 pgs: 129 active+clean; 327 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.6 MiB/s wr, 569 op/s 2024-03-18T00:00:40.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:39 smithi191 ceph-mon[36708]: pgmap v496: 129 pgs: 129 active+clean; 329 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.1 MiB/s wr, 526 op/s 2024-03-18T00:00:40.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:39 smithi012 ceph-mon[31541]: pgmap v496: 129 pgs: 129 active+clean; 329 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.1 MiB/s wr, 526 op/s 2024-03-18T00:00:40.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:39 smithi079 ceph-mon[36241]: pgmap v496: 129 pgs: 129 active+clean; 329 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.1 MiB/s wr, 526 op/s 2024-03-18T00:00:42.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:41 smithi012 ceph-mon[31541]: pgmap v497: 129 pgs: 129 active+clean; 329 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.0 MiB/s wr, 541 op/s 2024-03-18T00:00:42.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:41 smithi079 ceph-mon[36241]: pgmap v497: 129 pgs: 129 active+clean; 329 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.0 MiB/s wr, 541 op/s 2024-03-18T00:00:42.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:41 smithi191 ceph-mon[36708]: pgmap v497: 129 pgs: 129 active+clean; 329 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.0 MiB/s wr, 541 op/s 2024-03-18T00:00:44.178 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:43 smithi012 ceph-mon[31541]: pgmap v498: 129 pgs: 129 active+clean; 329 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 391 op/s 2024-03-18T00:00:44.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:43 smithi079 ceph-mon[36241]: pgmap v498: 129 pgs: 129 active+clean; 329 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 391 op/s 2024-03-18T00:00:44.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:43 smithi191 ceph-mon[36708]: pgmap v498: 129 pgs: 129 active+clean; 329 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 391 op/s 2024-03-18T00:00:44.558 INFO:tasks.workunit.client.0.smithi012.stdout: 78 501355 419 358416 554 276833 1268 2024-03-18T00:00:46.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:45 smithi012 ceph-mon[31541]: pgmap v499: 129 pgs: 129 active+clean; 331 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 376 op/s 2024-03-18T00:00:46.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:45 smithi079 ceph-mon[36241]: pgmap v499: 129 pgs: 129 active+clean; 331 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 376 op/s 2024-03-18T00:00:46.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:45 smithi191 ceph-mon[36708]: pgmap v499: 129 pgs: 129 active+clean; 331 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 376 op/s 2024-03-18T00:00:48.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:47 smithi012 ceph-mon[31541]: pgmap v500: 129 pgs: 129 active+clean; 334 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 236 op/s 2024-03-18T00:00:48.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:47 smithi079 ceph-mon[36241]: pgmap v500: 129 pgs: 129 active+clean; 334 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 236 op/s 2024-03-18T00:00:48.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:47 smithi191 ceph-mon[36708]: pgmap v500: 129 pgs: 129 active+clean; 334 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 236 op/s 2024-03-18T00:00:50.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:49 smithi012 ceph-mon[31541]: pgmap v501: 129 pgs: 129 active+clean; 334 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-03-18T00:00:50.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:49 smithi079 ceph-mon[36241]: pgmap v501: 129 pgs: 129 active+clean; 334 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-03-18T00:00:50.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:49 smithi191 ceph-mon[36708]: pgmap v501: 129 pgs: 129 active+clean; 334 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-03-18T00:00:52.179 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:51 smithi012 ceph-mon[31541]: pgmap v502: 129 pgs: 129 active+clean; 339 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 152 op/s 2024-03-18T00:00:52.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:51 smithi079 ceph-mon[36241]: pgmap v502: 129 pgs: 129 active+clean; 339 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 152 op/s 2024-03-18T00:00:52.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:51 smithi191 ceph-mon[36708]: pgmap v502: 129 pgs: 129 active+clean; 339 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 152 op/s 2024-03-18T00:00:54.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:53 smithi012 ceph-mon[31541]: pgmap v503: 129 pgs: 129 active+clean; 339 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-03-18T00:00:54.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:53 smithi079 ceph-mon[36241]: pgmap v503: 129 pgs: 129 active+clean; 339 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-03-18T00:00:54.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:53 smithi191 ceph-mon[36708]: pgmap v503: 129 pgs: 129 active+clean; 339 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-03-18T00:00:54.559 INFO:tasks.workunit.client.0.smithi012.stdout: 84 508524 476 366670 474 288602 1215 2024-03-18T00:00:56.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:55 smithi012 ceph-mon[31541]: pgmap v504: 129 pgs: 129 active+clean; 343 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 156 op/s 2024-03-18T00:00:56.298 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:55 smithi079 ceph-mon[36241]: pgmap v504: 129 pgs: 129 active+clean; 343 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 156 op/s 2024-03-18T00:00:56.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:55 smithi191 ceph-mon[36708]: pgmap v504: 129 pgs: 129 active+clean; 343 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 156 op/s 2024-03-18T00:00:57.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:56 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:00:57.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:56 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:00:57.424 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:56 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:00:58.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:57 smithi079 ceph-mon[36241]: pgmap v505: 129 pgs: 129 active+clean; 344 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 179 op/s 2024-03-18T00:00:58.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:57 smithi191 ceph-mon[36708]: pgmap v505: 129 pgs: 129 active+clean; 344 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 179 op/s 2024-03-18T00:00:58.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:57 smithi012 ceph-mon[31541]: pgmap v505: 129 pgs: 129 active+clean; 344 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 179 op/s 2024-03-18T00:01:00.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:00:59 smithi079 ceph-mon[36241]: pgmap v506: 129 pgs: 129 active+clean; 353 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 185 op/s 2024-03-18T00:01:00.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:00:59 smithi191 ceph-mon[36708]: pgmap v506: 129 pgs: 129 active+clean; 353 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 185 op/s 2024-03-18T00:01:00.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:00:59 smithi012 ceph-mon[31541]: pgmap v506: 129 pgs: 129 active+clean; 353 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 185 op/s 2024-03-18T00:01:01.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:00 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:01:01.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:00 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:01:01.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:00 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:01:02.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:01 smithi079 ceph-mon[36241]: pgmap v507: 129 pgs: 129 active+clean; 362 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 282 op/s 2024-03-18T00:01:02.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:01 smithi191 ceph-mon[36708]: pgmap v507: 129 pgs: 129 active+clean; 362 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 282 op/s 2024-03-18T00:01:02.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:01 smithi012 ceph-mon[31541]: pgmap v507: 129 pgs: 129 active+clean; 362 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 282 op/s 2024-03-18T00:01:04.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:03 smithi079 ceph-mon[36241]: pgmap v508: 129 pgs: 129 active+clean; 365 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 283 op/s 2024-03-18T00:01:04.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:03 smithi191 ceph-mon[36708]: pgmap v508: 129 pgs: 129 active+clean; 365 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 283 op/s 2024-03-18T00:01:04.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:03 smithi012 ceph-mon[31541]: pgmap v508: 129 pgs: 129 active+clean; 365 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 283 op/s 2024-03-18T00:01:04.558 INFO:tasks.workunit.client.0.smithi012.stdout: 88 399432 141 292977 191 222002 570 2024-03-18T00:01:06.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:05 smithi079 ceph-mon[36241]: pgmap v509: 129 pgs: 129 active+clean; 398 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.7 MiB/s wr, 452 op/s 2024-03-18T00:01:06.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:05 smithi191 ceph-mon[36708]: pgmap v509: 129 pgs: 129 active+clean; 398 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.7 MiB/s wr, 452 op/s 2024-03-18T00:01:06.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:05 smithi012 ceph-mon[31541]: pgmap v509: 129 pgs: 129 active+clean; 398 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.7 MiB/s wr, 452 op/s 2024-03-18T00:01:08.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:08 smithi079 ceph-mon[36241]: pgmap v510: 129 pgs: 129 active+clean; 426 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.7 MiB/s wr, 543 op/s 2024-03-18T00:01:08.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:08 smithi191 ceph-mon[36708]: pgmap v510: 129 pgs: 129 active+clean; 426 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.7 MiB/s wr, 543 op/s 2024-03-18T00:01:08.453 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:08 smithi012 ceph-mon[31541]: pgmap v510: 129 pgs: 129 active+clean; 426 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.7 MiB/s wr, 543 op/s 2024-03-18T00:01:10.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:10 smithi079 ceph-mon[36241]: pgmap v511: 129 pgs: 129 active+clean; 440 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 8.7 MiB/s wr, 565 op/s 2024-03-18T00:01:10.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:10 smithi191 ceph-mon[36708]: pgmap v511: 129 pgs: 129 active+clean; 440 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 8.7 MiB/s wr, 565 op/s 2024-03-18T00:01:10.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:10 smithi012 ceph-mon[31541]: pgmap v511: 129 pgs: 129 active+clean; 440 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 8.7 MiB/s wr, 565 op/s 2024-03-18T00:01:12.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:12 smithi079 ceph-mon[36241]: pgmap v512: 129 pgs: 129 active+clean; 438 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.9 MiB/s wr, 577 op/s 2024-03-18T00:01:12.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:12 smithi191 ceph-mon[36708]: pgmap v512: 129 pgs: 129 active+clean; 438 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.9 MiB/s wr, 577 op/s 2024-03-18T00:01:12.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:12 smithi012 ceph-mon[31541]: pgmap v512: 129 pgs: 129 active+clean; 438 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.9 MiB/s wr, 577 op/s 2024-03-18T00:01:14.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:14 smithi079 ceph-mon[36241]: pgmap v513: 129 pgs: 129 active+clean; 440 MiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.2 MiB/s wr, 487 op/s 2024-03-18T00:01:14.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:14 smithi191 ceph-mon[36708]: pgmap v513: 129 pgs: 129 active+clean; 440 MiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.2 MiB/s wr, 487 op/s 2024-03-18T00:01:14.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:14 smithi012 ceph-mon[31541]: pgmap v513: 129 pgs: 129 active+clean; 440 MiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.2 MiB/s wr, 487 op/s 2024-03-18T00:01:14.572 INFO:tasks.workunit.client.0.smithi012.stdout: 93 505943 465 367856 343 278495 1156 2024-03-18T00:01:16.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:16 smithi079 ceph-mon[36241]: pgmap v514: 129 pgs: 129 active+clean; 444 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.4 MiB/s wr, 479 op/s 2024-03-18T00:01:16.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:16 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:01:16.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:16 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:01:16.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:16 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:01:16.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:16 smithi191 ceph-mon[36708]: pgmap v514: 129 pgs: 129 active+clean; 444 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.4 MiB/s wr, 479 op/s 2024-03-18T00:01:16.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:16 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:01:16.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:16 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:01:16.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:16 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:01:16.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:16 smithi012 ceph-mon[31541]: pgmap v514: 129 pgs: 129 active+clean; 444 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.4 MiB/s wr, 479 op/s 2024-03-18T00:01:16.423 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:16 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:01:16.424 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:16 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:01:16.424 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:16 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:01:18.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:18 smithi191 ceph-mon[36708]: pgmap v515: 129 pgs: 129 active+clean; 443 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 307 op/s 2024-03-18T00:01:18.424 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:18 smithi012 ceph-mon[31541]: pgmap v515: 129 pgs: 129 active+clean; 443 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 307 op/s 2024-03-18T00:01:18.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:18 smithi079 ceph-mon[36241]: pgmap v515: 129 pgs: 129 active+clean; 443 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 307 op/s 2024-03-18T00:01:20.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:20 smithi191 ceph-mon[36708]: pgmap v516: 129 pgs: 129 active+clean; 443 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 190 op/s 2024-03-18T00:01:20.347 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:20 smithi079 ceph-mon[36241]: pgmap v516: 129 pgs: 129 active+clean; 443 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 190 op/s 2024-03-18T00:01:20.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:20 smithi012 ceph-mon[31541]: pgmap v516: 129 pgs: 129 active+clean; 443 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 190 op/s 2024-03-18T00:01:22.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:22 smithi191 ceph-mon[36708]: pgmap v517: 129 pgs: 129 active+clean; 445 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-03-18T00:01:22.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:22 smithi012 ceph-mon[31541]: pgmap v517: 129 pgs: 129 active+clean; 445 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-03-18T00:01:22.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:22 smithi079 ceph-mon[36241]: pgmap v517: 129 pgs: 129 active+clean; 445 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-03-18T00:01:24.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:24 smithi191 ceph-mon[36708]: pgmap v518: 129 pgs: 129 active+clean; 445 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-18T00:01:24.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:24 smithi012 ceph-mon[31541]: pgmap v518: 129 pgs: 129 active+clean; 445 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-18T00:01:24.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:24 smithi079 ceph-mon[36241]: pgmap v518: 129 pgs: 129 active+clean; 445 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-18T00:01:24.576 INFO:tasks.workunit.client.0.smithi012.stdout: 100 511891 397 379286 562 290035 1330 2024-03-18T00:01:26.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:26 smithi191 ceph-mon[36708]: pgmap v519: 129 pgs: 129 active+clean; 452 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-03-18T00:01:26.426 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:26 smithi012 ceph-mon[31541]: pgmap v519: 129 pgs: 129 active+clean; 452 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-03-18T00:01:26.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:26 smithi079 ceph-mon[36241]: pgmap v519: 129 pgs: 129 active+clean; 452 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-03-18T00:01:27.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:27 smithi191 ceph-mon[36708]: pgmap v520: 129 pgs: 129 active+clean; 451 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-03-18T00:01:27.425 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:27 smithi012 ceph-mon[31541]: pgmap v520: 129 pgs: 129 active+clean; 451 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-03-18T00:01:27.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:27 smithi079 ceph-mon[36241]: pgmap v520: 129 pgs: 129 active+clean; 451 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-03-18T00:01:28.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:01:28.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:01:28.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:28 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:01:28.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:01:28.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:01:28.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:28 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:01:28.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:01:28.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:01:28.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:28 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:01:29.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:29 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:01:29.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:29 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:01:29.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:29 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:01:29.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:29 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:01:29.545 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:29 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:01:29.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:29 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:01:29.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:29 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:01:29.546 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:29 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:01:29.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:29 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:01:29.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:29 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:01:29.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:29 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:01:29.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:29 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:01:33.681 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:33 smithi012 ceph-mon[31541]: pgmap v521: 129 pgs: 129 active+clean; 455 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 149 op/s 2024-03-18T00:01:33.681 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:33 smithi012 ceph-mon[31541]: pgmap v522: 129 pgs: 129 active+clean; 468 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 233 op/s 2024-03-18T00:01:33.681 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:33 smithi012 ceph-mon[31541]: pgmap v523: 129 pgs: 129 active+clean; 468 MiB data, 962 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 219 op/s 2024-03-18T00:01:33.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:33 smithi079 ceph-mon[36241]: pgmap v521: 129 pgs: 129 active+clean; 455 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 149 op/s 2024-03-18T00:01:33.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:33 smithi079 ceph-mon[36241]: pgmap v522: 129 pgs: 129 active+clean; 468 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 233 op/s 2024-03-18T00:01:33.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:33 smithi079 ceph-mon[36241]: pgmap v523: 129 pgs: 129 active+clean; 468 MiB data, 962 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 219 op/s 2024-03-18T00:01:33.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:33 smithi191 ceph-mon[36708]: pgmap v521: 129 pgs: 129 active+clean; 455 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 149 op/s 2024-03-18T00:01:33.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:33 smithi191 ceph-mon[36708]: pgmap v522: 129 pgs: 129 active+clean; 468 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 233 op/s 2024-03-18T00:01:33.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:33 smithi191 ceph-mon[36708]: pgmap v523: 129 pgs: 129 active+clean; 468 MiB data, 962 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 219 op/s 2024-03-18T00:01:34.595 INFO:tasks.workunit.client.0.smithi012.stdout: 102 432650 228 322158 178 245109 448 2024-03-18T00:01:35.673 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:35 smithi012 ceph-mon[31541]: pgmap v524: 129 pgs: 129 active+clean; 498 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 395 op/s 2024-03-18T00:01:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:35 smithi079 ceph-mon[36241]: pgmap v524: 129 pgs: 129 active+clean; 498 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 395 op/s 2024-03-18T00:01:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:35 smithi191 ceph-mon[36708]: pgmap v524: 129 pgs: 129 active+clean; 498 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 395 op/s 2024-03-18T00:01:37.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:37 smithi012 ceph-mon[31541]: pgmap v525: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.6 MiB/s wr, 490 op/s 2024-03-18T00:01:37.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:37 smithi079 ceph-mon[36241]: pgmap v525: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.6 MiB/s wr, 490 op/s 2024-03-18T00:01:37.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:37 smithi191 ceph-mon[36708]: pgmap v525: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.6 MiB/s wr, 490 op/s 2024-03-18T00:01:39.680 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:39 smithi012 ceph-mon[31541]: pgmap v526: 129 pgs: 129 active+clean; 532 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.5 MiB/s wr, 553 op/s 2024-03-18T00:01:39.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:39 smithi079 ceph-mon[36241]: pgmap v526: 129 pgs: 129 active+clean; 532 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.5 MiB/s wr, 553 op/s 2024-03-18T00:01:39.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:39 smithi191 ceph-mon[36708]: pgmap v526: 129 pgs: 129 active+clean; 532 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.5 MiB/s wr, 553 op/s 2024-03-18T00:01:41.643 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:41 smithi191 ceph-mon[36708]: pgmap v527: 129 pgs: 129 active+clean; 536 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.8 MiB/s wr, 566 op/s 2024-03-18T00:01:41.764 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:41 smithi012 ceph-mon[31541]: pgmap v527: 129 pgs: 129 active+clean; 536 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.8 MiB/s wr, 566 op/s 2024-03-18T00:01:41.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:41 smithi079 ceph-mon[36241]: pgmap v527: 129 pgs: 129 active+clean; 536 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.8 MiB/s wr, 566 op/s 2024-03-18T00:01:43.671 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:43 smithi012 ceph-mon[31541]: pgmap v528: 129 pgs: 129 active+clean; 536 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.5 MiB/s wr, 475 op/s 2024-03-18T00:01:43.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:43 smithi079 ceph-mon[36241]: pgmap v528: 129 pgs: 129 active+clean; 536 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.5 MiB/s wr, 475 op/s 2024-03-18T00:01:43.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:43 smithi191 ceph-mon[36708]: pgmap v528: 129 pgs: 129 active+clean; 536 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.5 MiB/s wr, 475 op/s 2024-03-18T00:01:44.599 INFO:tasks.workunit.client.0.smithi012.stdout: 110 475209 573 342438 304 266606 1076 2024-03-18T00:01:45.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:45 smithi012 ceph-mon[31541]: pgmap v529: 129 pgs: 129 active+clean; 538 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.8 MiB/s wr, 484 op/s 2024-03-18T00:01:45.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:45 smithi079 ceph-mon[36241]: pgmap v529: 129 pgs: 129 active+clean; 538 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.8 MiB/s wr, 484 op/s 2024-03-18T00:01:45.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:45 smithi191 ceph-mon[36708]: pgmap v529: 129 pgs: 129 active+clean; 538 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.8 MiB/s wr, 484 op/s 2024-03-18T00:01:47.671 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:47 smithi012 ceph-mon[31541]: pgmap v530: 129 pgs: 129 active+clean; 543 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.5 MiB/s wr, 317 op/s 2024-03-18T00:01:47.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:47 smithi079 ceph-mon[36241]: pgmap v530: 129 pgs: 129 active+clean; 543 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.5 MiB/s wr, 317 op/s 2024-03-18T00:01:47.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:47 smithi191 ceph-mon[36708]: pgmap v530: 129 pgs: 129 active+clean; 543 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.5 MiB/s wr, 317 op/s 2024-03-18T00:01:49.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:49 smithi079 ceph-mon[36241]: pgmap v531: 129 pgs: 129 active+clean; 544 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 205 op/s 2024-03-18T00:01:49.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:49 smithi191 ceph-mon[36708]: pgmap v531: 129 pgs: 129 active+clean; 544 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 205 op/s 2024-03-18T00:01:49.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:49 smithi012 ceph-mon[31541]: pgmap v531: 129 pgs: 129 active+clean; 544 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 205 op/s 2024-03-18T00:01:51.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:51 smithi191 ceph-mon[36708]: pgmap v532: 129 pgs: 129 active+clean; 543 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 155 op/s 2024-03-18T00:01:51.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:51 smithi012 ceph-mon[31541]: pgmap v532: 129 pgs: 129 active+clean; 543 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 155 op/s 2024-03-18T00:01:52.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:51 smithi079 ceph-mon[36241]: pgmap v532: 129 pgs: 129 active+clean; 543 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 155 op/s 2024-03-18T00:01:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:53 smithi191 ceph-mon[36708]: pgmap v533: 129 pgs: 129 active+clean; 545 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 103 op/s 2024-03-18T00:01:53.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:53 smithi012 ceph-mon[31541]: pgmap v533: 129 pgs: 129 active+clean; 545 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 103 op/s 2024-03-18T00:01:54.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:53 smithi079 ceph-mon[36241]: pgmap v533: 129 pgs: 129 active+clean; 545 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 103 op/s 2024-03-18T00:01:54.614 INFO:tasks.workunit.client.0.smithi012.stdout: 119 529168 383 375388 461 291042 1298 2024-03-18T00:01:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:55 smithi191 ceph-mon[36708]: pgmap v534: 129 pgs: 129 active+clean; 549 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 128 op/s 2024-03-18T00:01:55.921 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:55 smithi012 ceph-mon[31541]: pgmap v534: 129 pgs: 129 active+clean; 549 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 128 op/s 2024-03-18T00:01:56.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:55 smithi079 ceph-mon[36241]: pgmap v534: 129 pgs: 129 active+clean; 549 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 128 op/s 2024-03-18T00:01:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:57 smithi191 ceph-mon[36708]: pgmap v535: 129 pgs: 129 active+clean; 549 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 121 op/s 2024-03-18T00:01:57.924 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:57 smithi012 ceph-mon[31541]: pgmap v535: 129 pgs: 129 active+clean; 549 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 121 op/s 2024-03-18T00:01:58.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:57 smithi079 ceph-mon[36241]: pgmap v535: 129 pgs: 129 active+clean; 549 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 121 op/s 2024-03-18T00:01:59.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:01:59 smithi191 ceph-mon[36708]: pgmap v536: 129 pgs: 129 active+clean; 556 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 153 op/s 2024-03-18T00:01:59.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:01:59 smithi012 ceph-mon[31541]: pgmap v536: 129 pgs: 129 active+clean; 556 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 153 op/s 2024-03-18T00:02:00.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:01:59 smithi079 ceph-mon[36241]: pgmap v536: 129 pgs: 129 active+clean; 556 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 153 op/s 2024-03-18T00:02:00.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:00 smithi012 ceph-mon[31541]: pgmap v537: 129 pgs: 129 active+clean; 572 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 298 op/s 2024-03-18T00:02:01.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:00 smithi079 ceph-mon[36241]: pgmap v537: 129 pgs: 129 active+clean; 572 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 298 op/s 2024-03-18T00:02:01.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:00 smithi191 ceph-mon[36708]: pgmap v537: 129 pgs: 129 active+clean; 572 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 298 op/s 2024-03-18T00:02:02.923 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:02 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:02:02.923 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:02 smithi012 ceph-mon[31541]: pgmap v538: 129 pgs: 129 active+clean; 573 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 296 op/s 2024-03-18T00:02:03.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:02 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:02:03.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:02 smithi079 ceph-mon[36241]: pgmap v538: 129 pgs: 129 active+clean; 573 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 296 op/s 2024-03-18T00:02:03.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:02 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:02:03.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:02 smithi191 ceph-mon[36708]: pgmap v538: 129 pgs: 129 active+clean; 573 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 296 op/s 2024-03-18T00:02:04.607 INFO:tasks.workunit.client.0.smithi012.stdout: 124 403598 253 291097 211 236700 709 2024-03-18T00:02:05.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:05 smithi012 ceph-mon[31541]: pgmap v539: 129 pgs: 129 active+clean; 591 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 534 op/s 2024-03-18T00:02:05.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:05 smithi079 ceph-mon[36241]: pgmap v539: 129 pgs: 129 active+clean; 591 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 534 op/s 2024-03-18T00:02:05.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:05 smithi191 ceph-mon[36708]: pgmap v539: 129 pgs: 129 active+clean; 591 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 534 op/s 2024-03-18T00:02:07.681 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:07 smithi012 ceph-mon[31541]: pgmap v540: 129 pgs: 129 active+clean; 616 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 737 op/s 2024-03-18T00:02:07.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:07 smithi079 ceph-mon[36241]: pgmap v540: 129 pgs: 129 active+clean; 616 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 737 op/s 2024-03-18T00:02:07.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:07 smithi191 ceph-mon[36708]: pgmap v540: 129 pgs: 129 active+clean; 616 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 737 op/s 2024-03-18T00:02:09.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:09 smithi012 ceph-mon[31541]: pgmap v541: 129 pgs: 129 active+clean; 623 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.9 MiB/s wr, 842 op/s 2024-03-18T00:02:09.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:09 smithi079 ceph-mon[36241]: pgmap v541: 129 pgs: 129 active+clean; 623 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.9 MiB/s wr, 842 op/s 2024-03-18T00:02:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:09 smithi191 ceph-mon[36708]: pgmap v541: 129 pgs: 129 active+clean; 623 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.9 MiB/s wr, 842 op/s 2024-03-18T00:02:11.642 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:11 smithi191 ceph-mon[36708]: pgmap v542: 129 pgs: 129 active+clean; 640 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.8 MiB/s wr, 992 op/s 2024-03-18T00:02:11.684 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:11 smithi012 ceph-mon[31541]: pgmap v542: 129 pgs: 129 active+clean; 640 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.8 MiB/s wr, 992 op/s 2024-03-18T00:02:11.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:11 smithi079 ceph-mon[36241]: pgmap v542: 129 pgs: 129 active+clean; 640 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.8 MiB/s wr, 992 op/s 2024-03-18T00:02:13.670 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:13 smithi012 ceph-mon[31541]: pgmap v543: 129 pgs: 129 active+clean; 640 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 858 op/s 2024-03-18T00:02:13.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:13 smithi079 ceph-mon[36241]: pgmap v543: 129 pgs: 129 active+clean; 640 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 858 op/s 2024-03-18T00:02:13.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:13 smithi191 ceph-mon[36708]: pgmap v543: 129 pgs: 129 active+clean; 640 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 858 op/s 2024-03-18T00:02:14.629 INFO:tasks.workunit.client.0.smithi012.stdout: 131 370520 343 263917 455 207841 824 2024-03-18T00:02:15.677 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:15 smithi012 ceph-mon[31541]: pgmap v544: 129 pgs: 129 active+clean; 641 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.6 MiB/s wr, 976 op/s 2024-03-18T00:02:15.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:15 smithi079 ceph-mon[36241]: pgmap v544: 129 pgs: 129 active+clean; 641 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.6 MiB/s wr, 976 op/s 2024-03-18T00:02:15.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:15 smithi191 ceph-mon[36708]: pgmap v544: 129 pgs: 129 active+clean; 641 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.6 MiB/s wr, 976 op/s 2024-03-18T00:02:17.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:17 smithi012 ceph-mon[31541]: pgmap v545: 129 pgs: 129 active+clean; 640 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 842 op/s 2024-03-18T00:02:17.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:17 smithi079 ceph-mon[36241]: pgmap v545: 129 pgs: 129 active+clean; 640 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 842 op/s 2024-03-18T00:02:17.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:17 smithi191 ceph-mon[36708]: pgmap v545: 129 pgs: 129 active+clean; 640 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 842 op/s 2024-03-18T00:02:19.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:19 smithi012 ceph-mon[31541]: pgmap v546: 129 pgs: 129 active+clean; 637 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 645 op/s 2024-03-18T00:02:19.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:19 smithi079 ceph-mon[36241]: pgmap v546: 129 pgs: 129 active+clean; 637 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 645 op/s 2024-03-18T00:02:19.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:19 smithi191 ceph-mon[36708]: pgmap v546: 129 pgs: 129 active+clean; 637 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 645 op/s 2024-03-18T00:02:21.683 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:21 smithi012 ceph-mon[31541]: pgmap v547: 129 pgs: 129 active+clean; 615 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 608 op/s 2024-03-18T00:02:21.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:21 smithi079 ceph-mon[36241]: pgmap v547: 129 pgs: 129 active+clean; 615 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 608 op/s 2024-03-18T00:02:21.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:21 smithi191 ceph-mon[36708]: pgmap v547: 129 pgs: 129 active+clean; 615 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 608 op/s 2024-03-18T00:02:23.672 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:23 smithi012 ceph-mon[31541]: pgmap v548: 129 pgs: 129 active+clean; 619 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 417 op/s 2024-03-18T00:02:23.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:23 smithi079 ceph-mon[36241]: pgmap v548: 129 pgs: 129 active+clean; 619 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 417 op/s 2024-03-18T00:02:23.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:23 smithi191 ceph-mon[36708]: pgmap v548: 129 pgs: 129 active+clean; 619 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 417 op/s 2024-03-18T00:02:24.631 INFO:tasks.workunit.client.0.smithi012.stdout: 139 499850 390 357136 438 297441 1157 2024-03-18T00:02:25.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:25 smithi012 ceph-mon[31541]: pgmap v549: 129 pgs: 129 active+clean; 622 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 469 op/s 2024-03-18T00:02:25.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:25 smithi079 ceph-mon[36241]: pgmap v549: 129 pgs: 129 active+clean; 622 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 469 op/s 2024-03-18T00:02:25.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:25 smithi191 ceph-mon[36708]: pgmap v549: 129 pgs: 129 active+clean; 622 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 469 op/s 2024-03-18T00:02:27.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:27 smithi079 ceph-mon[36241]: pgmap v550: 129 pgs: 129 active+clean; 622 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 383 op/s 2024-03-18T00:02:27.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:27 smithi191 ceph-mon[36708]: pgmap v550: 129 pgs: 129 active+clean; 622 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 383 op/s 2024-03-18T00:02:27.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:27 smithi012 ceph-mon[31541]: pgmap v550: 129 pgs: 129 active+clean; 622 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 383 op/s 2024-03-18T00:02:29.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:29 smithi079 ceph-mon[36241]: pgmap v551: 129 pgs: 129 active+clean; 621 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 291 op/s 2024-03-18T00:02:29.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:29 smithi191 ceph-mon[36708]: pgmap v551: 129 pgs: 129 active+clean; 621 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 291 op/s 2024-03-18T00:02:29.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:29 smithi012 ceph-mon[31541]: pgmap v551: 129 pgs: 129 active+clean; 621 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 291 op/s 2024-03-18T00:02:30.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:30 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:02:30.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:30 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:02:30.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:30 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:02:31.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:31 smithi079 ceph-mon[36241]: pgmap v552: 129 pgs: 129 active+clean; 641 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 388 op/s 2024-03-18T00:02:31.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:31 smithi191 ceph-mon[36708]: pgmap v552: 129 pgs: 129 active+clean; 641 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 388 op/s 2024-03-18T00:02:31.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:31 smithi012 ceph-mon[31541]: pgmap v552: 129 pgs: 129 active+clean; 641 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 388 op/s 2024-03-18T00:02:33.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:33 smithi079 ceph-mon[36241]: pgmap v553: 129 pgs: 129 active+clean; 643 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 342 op/s 2024-03-18T00:02:33.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:33 smithi191 ceph-mon[36708]: pgmap v553: 129 pgs: 129 active+clean; 643 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 342 op/s 2024-03-18T00:02:33.921 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:33 smithi012 ceph-mon[31541]: pgmap v553: 129 pgs: 129 active+clean; 643 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 342 op/s 2024-03-18T00:02:34.637 INFO:tasks.workunit.client.0.smithi012.stdout: 142 411625 290 292857 243 246992 753 2024-03-18T00:02:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:35 smithi079 ceph-mon[36241]: pgmap v554: 129 pgs: 129 active+clean; 658 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 473 op/s 2024-03-18T00:02:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:35 smithi191 ceph-mon[36708]: pgmap v554: 129 pgs: 129 active+clean; 658 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 473 op/s 2024-03-18T00:02:35.930 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:35 smithi012 ceph-mon[31541]: pgmap v554: 129 pgs: 129 active+clean; 658 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 473 op/s 2024-03-18T00:02:37.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:37 smithi079 ceph-mon[36241]: pgmap v555: 129 pgs: 129 active+clean; 677 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.4 MiB/s wr, 577 op/s 2024-03-18T00:02:37.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:37 smithi191 ceph-mon[36708]: pgmap v555: 129 pgs: 129 active+clean; 677 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.4 MiB/s wr, 577 op/s 2024-03-18T00:02:37.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:37 smithi012 ceph-mon[31541]: pgmap v555: 129 pgs: 129 active+clean; 677 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.4 MiB/s wr, 577 op/s 2024-03-18T00:02:39.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:39 smithi079 ceph-mon[36241]: pgmap v556: 129 pgs: 129 active+clean; 687 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.2 MiB/s wr, 629 op/s 2024-03-18T00:02:39.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:39 smithi191 ceph-mon[36708]: pgmap v556: 129 pgs: 129 active+clean; 687 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.2 MiB/s wr, 629 op/s 2024-03-18T00:02:39.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:39 smithi012 ceph-mon[31541]: pgmap v556: 129 pgs: 129 active+clean; 687 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.2 MiB/s wr, 629 op/s 2024-03-18T00:02:41.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:41 smithi079 ceph-mon[36241]: pgmap v557: 129 pgs: 129 active+clean; 702 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.5 MiB/s wr, 751 op/s 2024-03-18T00:02:41.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:41 smithi191 ceph-mon[36708]: pgmap v557: 129 pgs: 129 active+clean; 702 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.5 MiB/s wr, 751 op/s 2024-03-18T00:02:41.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:41 smithi012 ceph-mon[31541]: pgmap v557: 129 pgs: 129 active+clean; 702 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.5 MiB/s wr, 751 op/s 2024-03-18T00:02:43.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:43 smithi079 ceph-mon[36241]: pgmap v558: 129 pgs: 129 active+clean; 698 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.5 MiB/s wr, 720 op/s 2024-03-18T00:02:43.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:43 smithi191 ceph-mon[36708]: pgmap v558: 129 pgs: 129 active+clean; 698 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.5 MiB/s wr, 720 op/s 2024-03-18T00:02:43.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:43 smithi012 ceph-mon[31541]: pgmap v558: 129 pgs: 129 active+clean; 698 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.5 MiB/s wr, 720 op/s 2024-03-18T00:02:44.637 INFO:tasks.workunit.client.0.smithi012.stdout: 149 425340 398 302539 232 245236 889 2024-03-18T00:02:45.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:45 smithi079 ceph-mon[36241]: pgmap v559: 129 pgs: 129 active+clean; 687 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.4 MiB/s wr, 745 op/s 2024-03-18T00:02:45.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:45 smithi191 ceph-mon[36708]: pgmap v559: 129 pgs: 129 active+clean; 687 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.4 MiB/s wr, 745 op/s 2024-03-18T00:02:45.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:45 smithi012 ceph-mon[31541]: pgmap v559: 129 pgs: 129 active+clean; 687 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.4 MiB/s wr, 745 op/s 2024-03-18T00:02:47.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:47 smithi079 ceph-mon[36241]: pgmap v560: 129 pgs: 129 active+clean; 682 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 668 op/s 2024-03-18T00:02:47.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:47 smithi191 ceph-mon[36708]: pgmap v560: 129 pgs: 129 active+clean; 682 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 668 op/s 2024-03-18T00:02:47.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:47 smithi012 ceph-mon[31541]: pgmap v560: 129 pgs: 129 active+clean; 682 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 668 op/s 2024-03-18T00:02:49.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:49 smithi079 ceph-mon[36241]: pgmap v561: 129 pgs: 129 active+clean; 684 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 511 op/s 2024-03-18T00:02:49.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:49 smithi191 ceph-mon[36708]: pgmap v561: 129 pgs: 129 active+clean; 684 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 511 op/s 2024-03-18T00:02:49.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:49 smithi012 ceph-mon[31541]: pgmap v561: 129 pgs: 129 active+clean; 684 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 511 op/s 2024-03-18T00:02:51.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:51 smithi079 ceph-mon[36241]: pgmap v562: 129 pgs: 129 active+clean; 685 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 434 op/s 2024-03-18T00:02:51.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:51 smithi191 ceph-mon[36708]: pgmap v562: 129 pgs: 129 active+clean; 685 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 434 op/s 2024-03-18T00:02:51.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:51 smithi012 ceph-mon[31541]: pgmap v562: 129 pgs: 129 active+clean; 685 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 434 op/s 2024-03-18T00:02:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:53 smithi191 ceph-mon[36708]: pgmap v563: 129 pgs: 129 active+clean; 687 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 317 op/s 2024-03-18T00:02:53.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:53 smithi012 ceph-mon[31541]: pgmap v563: 129 pgs: 129 active+clean; 687 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 317 op/s 2024-03-18T00:02:54.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:53 smithi079 ceph-mon[36241]: pgmap v563: 129 pgs: 129 active+clean; 687 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 317 op/s 2024-03-18T00:02:54.639 INFO:tasks.workunit.client.0.smithi012.stdout: 154 504015 471 350997 379 303518 1344 2024-03-18T00:02:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:55 smithi191 ceph-mon[36708]: pgmap v564: 129 pgs: 129 active+clean; 686 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 251 op/s 2024-03-18T00:02:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:02:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:02:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:55 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:02:55.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:55 smithi012 ceph-mon[31541]: pgmap v564: 129 pgs: 129 active+clean; 686 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 251 op/s 2024-03-18T00:02:55.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:02:55.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:02:55.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:55 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:02:56.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:55 smithi079 ceph-mon[36241]: pgmap v564: 129 pgs: 129 active+clean; 686 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 251 op/s 2024-03-18T00:02:56.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:02:56.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:02:56.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:55 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:02:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:57 smithi191 ceph-mon[36708]: pgmap v565: 129 pgs: 129 active+clean; 690 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 234 op/s 2024-03-18T00:02:57.922 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:57 smithi012 ceph-mon[31541]: pgmap v565: 129 pgs: 129 active+clean; 690 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 234 op/s 2024-03-18T00:02:58.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:57 smithi079 ceph-mon[36241]: pgmap v565: 129 pgs: 129 active+clean; 690 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 234 op/s 2024-03-18T00:02:59.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:02:59 smithi191 ceph-mon[36708]: pgmap v566: 129 pgs: 129 active+clean; 693 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 191 op/s 2024-03-18T00:02:59.926 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:02:59 smithi012 ceph-mon[31541]: pgmap v566: 129 pgs: 129 active+clean; 693 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 191 op/s 2024-03-18T00:03:00.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:02:59 smithi079 ceph-mon[36241]: pgmap v566: 129 pgs: 129 active+clean; 693 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 191 op/s 2024-03-18T00:03:01.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:01 smithi191 ceph-mon[36708]: pgmap v567: 129 pgs: 129 active+clean; 694 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 205 op/s 2024-03-18T00:03:01.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:01 smithi012 ceph-mon[31541]: pgmap v567: 129 pgs: 129 active+clean; 694 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 205 op/s 2024-03-18T00:03:02.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:01 smithi079 ceph-mon[36241]: pgmap v567: 129 pgs: 129 active+clean; 694 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 205 op/s 2024-03-18T00:03:03.926 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:03 smithi012 ceph-mon[31541]: pgmap v568: 129 pgs: 129 active+clean; 696 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 214 op/s 2024-03-18T00:03:04.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:03 smithi079 ceph-mon[36241]: pgmap v568: 129 pgs: 129 active+clean; 696 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 214 op/s 2024-03-18T00:03:04.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:03 smithi191 ceph-mon[36708]: pgmap v568: 129 pgs: 129 active+clean; 696 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 214 op/s 2024-03-18T00:03:04.645 INFO:tasks.workunit.client.0.smithi012.stdout: 158 426136 292 297802 218 256223 834 2024-03-18T00:03:05.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:05 smithi012 ceph-mon[31541]: pgmap v569: 129 pgs: 129 active+clean; 703 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 396 op/s 2024-03-18T00:03:06.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:05 smithi079 ceph-mon[36241]: pgmap v569: 129 pgs: 129 active+clean; 703 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 396 op/s 2024-03-18T00:03:06.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:05 smithi191 ceph-mon[36708]: pgmap v569: 129 pgs: 129 active+clean; 703 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 396 op/s 2024-03-18T00:03:07.929 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:07 smithi012 ceph-mon[31541]: pgmap v570: 129 pgs: 129 active+clean; 717 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.6 MiB/s wr, 499 op/s 2024-03-18T00:03:08.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:07 smithi079 ceph-mon[36241]: pgmap v570: 129 pgs: 129 active+clean; 717 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.6 MiB/s wr, 499 op/s 2024-03-18T00:03:08.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:07 smithi191 ceph-mon[36708]: pgmap v570: 129 pgs: 129 active+clean; 717 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.6 MiB/s wr, 499 op/s 2024-03-18T00:03:09.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:09 smithi012 ceph-mon[31541]: pgmap v571: 129 pgs: 129 active+clean; 730 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.9 MiB/s wr, 629 op/s 2024-03-18T00:03:10.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:09 smithi079 ceph-mon[36241]: pgmap v571: 129 pgs: 129 active+clean; 730 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.9 MiB/s wr, 629 op/s 2024-03-18T00:03:10.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:09 smithi191 ceph-mon[36708]: pgmap v571: 129 pgs: 129 active+clean; 730 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.9 MiB/s wr, 629 op/s 2024-03-18T00:03:11.893 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:11 smithi191 ceph-mon[36708]: pgmap v572: 129 pgs: 129 active+clean; 747 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 7.1 MiB/s wr, 764 op/s 2024-03-18T00:03:11.954 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:11 smithi012 ceph-mon[31541]: pgmap v572: 129 pgs: 129 active+clean; 747 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 7.1 MiB/s wr, 764 op/s 2024-03-18T00:03:12.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:11 smithi079 ceph-mon[36241]: pgmap v572: 129 pgs: 129 active+clean; 747 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 7.1 MiB/s wr, 764 op/s 2024-03-18T00:03:13.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:12 smithi079 ceph-mon[36241]: pgmap v573: 129 pgs: 129 active+clean; 749 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 7.1 MiB/s wr, 775 op/s 2024-03-18T00:03:13.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:12 smithi191 ceph-mon[36708]: pgmap v573: 129 pgs: 129 active+clean; 749 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 7.1 MiB/s wr, 775 op/s 2024-03-18T00:03:13.172 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:12 smithi012 ceph-mon[31541]: pgmap v573: 129 pgs: 129 active+clean; 749 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 7.1 MiB/s wr, 775 op/s 2024-03-18T00:03:14.646 INFO:tasks.workunit.client.0.smithi012.stdout: 162 383905 345 267745 200 237016 930 2024-03-18T00:03:15.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:15 smithi012 ceph-mon[31541]: pgmap v574: 129 pgs: 129 active+clean; 749 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.5 MiB/s wr, 869 op/s 2024-03-18T00:03:15.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:15 smithi079 ceph-mon[36241]: pgmap v574: 129 pgs: 129 active+clean; 749 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.5 MiB/s wr, 869 op/s 2024-03-18T00:03:15.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:15 smithi191 ceph-mon[36708]: pgmap v574: 129 pgs: 129 active+clean; 749 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.5 MiB/s wr, 869 op/s 2024-03-18T00:03:17.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:17 smithi012 ceph-mon[31541]: pgmap v575: 129 pgs: 129 active+clean; 751 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 716 op/s 2024-03-18T00:03:17.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:17 smithi079 ceph-mon[36241]: pgmap v575: 129 pgs: 129 active+clean; 751 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 716 op/s 2024-03-18T00:03:17.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:17 smithi191 ceph-mon[36708]: pgmap v575: 129 pgs: 129 active+clean; 751 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 716 op/s 2024-03-18T00:03:19.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:19 smithi012 ceph-mon[31541]: pgmap v576: 129 pgs: 129 active+clean; 753 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 664 op/s 2024-03-18T00:03:19.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:19 smithi079 ceph-mon[36241]: pgmap v576: 129 pgs: 129 active+clean; 753 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 664 op/s 2024-03-18T00:03:19.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:19 smithi191 ceph-mon[36708]: pgmap v576: 129 pgs: 129 active+clean; 753 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 664 op/s 2024-03-18T00:03:21.671 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:21 smithi012 ceph-mon[31541]: pgmap v577: 129 pgs: 129 active+clean; 752 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 565 op/s 2024-03-18T00:03:21.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:21 smithi079 ceph-mon[36241]: pgmap v577: 129 pgs: 129 active+clean; 752 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 565 op/s 2024-03-18T00:03:21.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:21 smithi191 ceph-mon[36708]: pgmap v577: 129 pgs: 129 active+clean; 752 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 565 op/s 2024-03-18T00:03:22.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:22 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:03:22.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:22 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:03:22.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:22 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:03:23.674 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:23 smithi012 ceph-mon[31541]: pgmap v578: 129 pgs: 129 active+clean; 752 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 429 op/s 2024-03-18T00:03:23.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:23 smithi079 ceph-mon[36241]: pgmap v578: 129 pgs: 129 active+clean; 752 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 429 op/s 2024-03-18T00:03:23.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:23 smithi191 ceph-mon[36708]: pgmap v578: 129 pgs: 129 active+clean; 752 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 429 op/s 2024-03-18T00:03:24.697 INFO:tasks.workunit.client.0.smithi012.stdout: 166 488784 495 339476 266 300792 1247 2024-03-18T00:03:25.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:25 smithi079 ceph-mon[36241]: pgmap v579: 129 pgs: 129 active+clean; 756 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 433 op/s 2024-03-18T00:03:25.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:25 smithi191 ceph-mon[36708]: pgmap v579: 129 pgs: 129 active+clean; 756 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 433 op/s 2024-03-18T00:03:25.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:25 smithi012 ceph-mon[31541]: pgmap v579: 129 pgs: 129 active+clean; 756 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 433 op/s 2024-03-18T00:03:27.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:27 smithi079 ceph-mon[36241]: pgmap v580: 129 pgs: 129 active+clean; 756 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 341 op/s 2024-03-18T00:03:27.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:27 smithi191 ceph-mon[36708]: pgmap v580: 129 pgs: 129 active+clean; 756 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 341 op/s 2024-03-18T00:03:27.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:27 smithi012 ceph-mon[31541]: pgmap v580: 129 pgs: 129 active+clean; 756 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 341 op/s 2024-03-18T00:03:29.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:29 smithi079 ceph-mon[36241]: pgmap v581: 129 pgs: 129 active+clean; 739 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1.1 MiB/s wr, 350 op/s 2024-03-18T00:03:29.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:29 smithi191 ceph-mon[36708]: pgmap v581: 129 pgs: 129 active+clean; 739 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1.1 MiB/s wr, 350 op/s 2024-03-18T00:03:29.921 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:29 smithi012 ceph-mon[31541]: pgmap v581: 129 pgs: 129 active+clean; 739 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1.1 MiB/s wr, 350 op/s 2024-03-18T00:03:31.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:31 smithi079 ceph-mon[36241]: pgmap v582: 129 pgs: 129 active+clean; 732 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 329 op/s 2024-03-18T00:03:31.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:31 smithi191 ceph-mon[36708]: pgmap v582: 129 pgs: 129 active+clean; 732 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 329 op/s 2024-03-18T00:03:31.927 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:31 smithi012 ceph-mon[31541]: pgmap v582: 129 pgs: 129 active+clean; 732 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 329 op/s 2024-03-18T00:03:33.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:33 smithi079 ceph-mon[36241]: pgmap v583: 129 pgs: 129 active+clean; 731 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 263 op/s 2024-03-18T00:03:33.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:33 smithi191 ceph-mon[36708]: pgmap v583: 129 pgs: 129 active+clean; 731 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 263 op/s 2024-03-18T00:03:33.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:33 smithi012 ceph-mon[31541]: pgmap v583: 129 pgs: 129 active+clean; 731 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 263 op/s 2024-03-18T00:03:34.700 INFO:tasks.workunit.client.0.smithi012.stdout: 170 451169 301 307324 243 275416 793 2024-03-18T00:03:35.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:35 smithi079 ceph-mon[36241]: pgmap v584: 129 pgs: 129 active+clean; 750 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 387 op/s 2024-03-18T00:03:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:35 smithi191 ceph-mon[36708]: pgmap v584: 129 pgs: 129 active+clean; 750 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 387 op/s 2024-03-18T00:03:35.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:35 smithi012 ceph-mon[31541]: pgmap v584: 129 pgs: 129 active+clean; 750 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 387 op/s 2024-03-18T00:03:37.722 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:37 smithi012 ceph-mon[31541]: pgmap v585: 129 pgs: 129 active+clean; 766 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 442 op/s 2024-03-18T00:03:37.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:37 smithi079 ceph-mon[36241]: pgmap v585: 129 pgs: 129 active+clean; 766 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 442 op/s 2024-03-18T00:03:37.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:37 smithi191 ceph-mon[36708]: pgmap v585: 129 pgs: 129 active+clean; 766 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 442 op/s 2024-03-18T00:03:39.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:39 smithi079 ceph-mon[36241]: pgmap v586: 129 pgs: 129 active+clean; 782 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 507 op/s 2024-03-18T00:03:39.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:39 smithi191 ceph-mon[36708]: pgmap v586: 129 pgs: 129 active+clean; 782 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 507 op/s 2024-03-18T00:03:39.924 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:39 smithi012 ceph-mon[31541]: pgmap v586: 129 pgs: 129 active+clean; 782 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 507 op/s 2024-03-18T00:03:41.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:41 smithi079 ceph-mon[36241]: pgmap v587: 129 pgs: 129 active+clean; 799 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.7 MiB/s wr, 616 op/s 2024-03-18T00:03:41.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:41 smithi191 ceph-mon[36708]: pgmap v587: 129 pgs: 129 active+clean; 799 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.7 MiB/s wr, 616 op/s 2024-03-18T00:03:41.925 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:41 smithi012 ceph-mon[31541]: pgmap v587: 129 pgs: 129 active+clean; 799 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.7 MiB/s wr, 616 op/s 2024-03-18T00:03:43.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:43 smithi079 ceph-mon[36241]: pgmap v588: 129 pgs: 129 active+clean; 802 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.4 MiB/s wr, 592 op/s 2024-03-18T00:03:43.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:43 smithi191 ceph-mon[36708]: pgmap v588: 129 pgs: 129 active+clean; 802 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.4 MiB/s wr, 592 op/s 2024-03-18T00:03:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:43 smithi012 ceph-mon[31541]: pgmap v588: 129 pgs: 129 active+clean; 802 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.4 MiB/s wr, 592 op/s 2024-03-18T00:03:44.738 INFO:tasks.workunit.client.0.smithi012.stdout: 175 397119 322 274627 290 247450 791 2024-03-18T00:03:45.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:45 smithi079 ceph-mon[36241]: pgmap v589: 129 pgs: 129 active+clean; 804 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.7 MiB/s wr, 672 op/s 2024-03-18T00:03:45.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:45 smithi191 ceph-mon[36708]: pgmap v589: 129 pgs: 129 active+clean; 804 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.7 MiB/s wr, 672 op/s 2024-03-18T00:03:45.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:45 smithi012 ceph-mon[31541]: pgmap v589: 129 pgs: 129 active+clean; 804 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.7 MiB/s wr, 672 op/s 2024-03-18T00:03:47.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:47 smithi079 ceph-mon[36241]: pgmap v590: 129 pgs: 129 active+clean; 812 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.8 MiB/s wr, 618 op/s 2024-03-18T00:03:47.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:47 smithi191 ceph-mon[36708]: pgmap v590: 129 pgs: 129 active+clean; 812 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.8 MiB/s wr, 618 op/s 2024-03-18T00:03:47.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:47 smithi012 ceph-mon[31541]: pgmap v590: 129 pgs: 129 active+clean; 812 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.8 MiB/s wr, 618 op/s 2024-03-18T00:03:49.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:49 smithi079 ceph-mon[36241]: pgmap v591: 129 pgs: 129 active+clean; 811 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 583 op/s 2024-03-18T00:03:49.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:49 smithi191 ceph-mon[36708]: pgmap v591: 129 pgs: 129 active+clean; 811 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 583 op/s 2024-03-18T00:03:49.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:49 smithi012 ceph-mon[31541]: pgmap v591: 129 pgs: 129 active+clean; 811 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 583 op/s 2024-03-18T00:03:51.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:51 smithi079 ceph-mon[36241]: pgmap v592: 129 pgs: 129 active+clean; 812 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 560 op/s 2024-03-18T00:03:51.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:51 smithi191 ceph-mon[36708]: pgmap v592: 129 pgs: 129 active+clean; 812 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 560 op/s 2024-03-18T00:03:51.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:51 smithi012 ceph-mon[31541]: pgmap v592: 129 pgs: 129 active+clean; 812 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 560 op/s 2024-03-18T00:03:53.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:53 smithi079 ceph-mon[36241]: pgmap v593: 129 pgs: 129 active+clean; 812 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 423 op/s 2024-03-18T00:03:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:53 smithi191 ceph-mon[36708]: pgmap v593: 129 pgs: 129 active+clean; 812 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 423 op/s 2024-03-18T00:03:53.930 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:53 smithi012 ceph-mon[31541]: pgmap v593: 129 pgs: 129 active+clean; 812 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 423 op/s 2024-03-18T00:03:54.731 INFO:tasks.workunit.client.0.smithi012.stdout: 182 444797 502 300946 305 275410 1252 2024-03-18T00:03:55.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:55 smithi079 ceph-mon[36241]: pgmap v594: 129 pgs: 129 active+clean; 799 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 445 op/s 2024-03-18T00:03:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:55 smithi191 ceph-mon[36708]: pgmap v594: 129 pgs: 129 active+clean; 799 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 445 op/s 2024-03-18T00:03:55.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:55 smithi012 ceph-mon[31541]: pgmap v594: 129 pgs: 129 active+clean; 799 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 445 op/s 2024-03-18T00:03:56.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:56 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:03:56.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:56 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:03:56.923 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:56 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:03:57.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:57 smithi079 ceph-mon[36241]: pgmap v595: 129 pgs: 129 active+clean; 791 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 421 op/s 2024-03-18T00:03:57.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:03:57.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:03:57.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:03:57.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:57 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:03:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:57 smithi191 ceph-mon[36708]: pgmap v595: 129 pgs: 129 active+clean; 791 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 421 op/s 2024-03-18T00:03:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:03:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:03:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:03:57.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:57 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:03:57.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:57 smithi012 ceph-mon[31541]: pgmap v595: 129 pgs: 129 active+clean; 791 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 421 op/s 2024-03-18T00:03:57.921 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:03:57.921 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:03:57.921 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:03:57.921 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:57 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:03:58.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:58 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi079 to 174.1M 2024-03-18T00:03:58.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:58 smithi191 ceph-mon[36708]: Unable to set osd_memory_target on smithi079 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-18T00:03:58.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:58 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi079 to 174.1M 2024-03-18T00:03:58.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:58 smithi012 ceph-mon[31541]: Unable to set osd_memory_target on smithi079 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-18T00:03:59.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:58 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi079 to 174.1M 2024-03-18T00:03:59.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:58 smithi079 ceph-mon[36241]: Unable to set osd_memory_target on smithi079 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-18T00:03:59.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:03:59 smithi191 ceph-mon[36708]: pgmap v596: 129 pgs: 129 active+clean; 788 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 397 op/s 2024-03-18T00:03:59.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:03:59 smithi012 ceph-mon[31541]: pgmap v596: 129 pgs: 129 active+clean; 788 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 397 op/s 2024-03-18T00:04:00.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:03:59 smithi079 ceph-mon[36241]: pgmap v596: 129 pgs: 129 active+clean; 788 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 397 op/s 2024-03-18T00:04:01.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:01 smithi191 ceph-mon[36708]: pgmap v597: 129 pgs: 129 active+clean; 788 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 409 op/s 2024-03-18T00:04:01.948 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:01 smithi012 ceph-mon[31541]: pgmap v597: 129 pgs: 129 active+clean; 788 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 409 op/s 2024-03-18T00:04:02.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:01 smithi079 ceph-mon[36241]: pgmap v597: 129 pgs: 129 active+clean; 788 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 409 op/s 2024-03-18T00:04:03.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:03 smithi191 ceph-mon[36708]: pgmap v598: 129 pgs: 129 active+clean; 788 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.1 MiB/s wr, 349 op/s 2024-03-18T00:04:03.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:03 smithi012 ceph-mon[31541]: pgmap v598: 129 pgs: 129 active+clean; 788 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.1 MiB/s wr, 349 op/s 2024-03-18T00:04:04.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:03 smithi079 ceph-mon[36241]: pgmap v598: 129 pgs: 129 active+clean; 788 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.1 MiB/s wr, 349 op/s 2024-03-18T00:04:04.746 INFO:tasks.workunit.client.0.smithi012.stdout: 185 432501 357 294870 219 265129 956 2024-03-18T00:04:05.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:05 smithi191 ceph-mon[36708]: pgmap v599: 129 pgs: 129 active+clean; 797 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 416 op/s 2024-03-18T00:04:05.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:05 smithi012 ceph-mon[31541]: pgmap v599: 129 pgs: 129 active+clean; 797 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 416 op/s 2024-03-18T00:04:06.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:05 smithi079 ceph-mon[36241]: pgmap v599: 129 pgs: 129 active+clean; 797 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 416 op/s 2024-03-18T00:04:06.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:04:06.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:04:06.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:04:06.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:04:06.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:06 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi012 to 177.8M 2024-03-18T00:04:06.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:06 smithi191 ceph-mon[36708]: Unable to set osd_memory_target on smithi012 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-18T00:04:06.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:04:06.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:04:06.842 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:04:06.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:04:06.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:04:06.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:04:06.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:04:06.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:06 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi012 to 177.8M 2024-03-18T00:04:06.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:06 smithi012 ceph-mon[31541]: Unable to set osd_memory_target on smithi012 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-18T00:04:06.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:04:06.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:04:06.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:04:07.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:04:07.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:04:07.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:04:07.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:04:07.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:06 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi012 to 177.8M 2024-03-18T00:04:07.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:06 smithi079 ceph-mon[36241]: Unable to set osd_memory_target on smithi012 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-18T00:04:07.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:04:07.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:04:07.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:04:08.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:07 smithi079 ceph-mon[36241]: pgmap v600: 129 pgs: 129 active+clean; 813 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 498 op/s 2024-03-18T00:04:08.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:07 smithi191 ceph-mon[36708]: pgmap v600: 129 pgs: 129 active+clean; 813 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 498 op/s 2024-03-18T00:04:08.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:07 smithi012 ceph-mon[31541]: pgmap v600: 129 pgs: 129 active+clean; 813 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 498 op/s 2024-03-18T00:04:09.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:08 smithi079 ceph-mon[36241]: pgmap v601: 129 pgs: 129 active+clean; 830 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 4.3 MiB/s wr, 556 op/s 2024-03-18T00:04:09.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:08 smithi191 ceph-mon[36708]: pgmap v601: 129 pgs: 129 active+clean; 830 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 4.3 MiB/s wr, 556 op/s 2024-03-18T00:04:09.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:08 smithi012 ceph-mon[31541]: pgmap v601: 129 pgs: 129 active+clean; 830 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 4.3 MiB/s wr, 556 op/s 2024-03-18T00:04:10.298 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-03-18T00:04:10.298 INFO:tasks.workunit.client.0.smithi012.stdout:Final score for writes: 185 2024-03-18T00:04:10.299 INFO:tasks.workunit.client.0.smithi012.stdout:Final score for reads : 274674 2024-03-18T00:04:10.299 INFO:tasks.workunit.client.0.smithi012.stdout: 2024-03-18T00:04:10.301 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-18T00:04:10.301 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-18T00:04:11.794 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:11 smithi191 ceph-mon[36708]: pgmap v602: 129 pgs: 129 active+clean; 858 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.8 MiB/s wr, 735 op/s 2024-03-18T00:04:11.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:11 smithi079 ceph-mon[36241]: pgmap v602: 129 pgs: 129 active+clean; 858 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.8 MiB/s wr, 735 op/s 2024-03-18T00:04:11.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:11 smithi012 ceph-mon[31541]: pgmap v602: 129 pgs: 129 active+clean; 858 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.8 MiB/s wr, 735 op/s 2024-03-18T00:04:13.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:13 smithi079 ceph-mon[36241]: pgmap v603: 129 pgs: 129 active+clean; 862 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.0 MiB/s wr, 692 op/s 2024-03-18T00:04:13.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:13 smithi191 ceph-mon[36708]: pgmap v603: 129 pgs: 129 active+clean; 862 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.0 MiB/s wr, 692 op/s 2024-03-18T00:04:13.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:13 smithi012 ceph-mon[31541]: pgmap v603: 129 pgs: 129 active+clean; 862 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.0 MiB/s wr, 692 op/s 2024-03-18T00:04:15.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:15 smithi012 ceph-mon[31541]: pgmap v604: 129 pgs: 129 active+clean; 862 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.5 MiB/s wr, 741 op/s 2024-03-18T00:04:16.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:15 smithi079 ceph-mon[36241]: pgmap v604: 129 pgs: 129 active+clean; 862 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.5 MiB/s wr, 741 op/s 2024-03-18T00:04:16.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:15 smithi191 ceph-mon[36708]: pgmap v604: 129 pgs: 129 active+clean; 862 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.5 MiB/s wr, 741 op/s 2024-03-18T00:04:17.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:17 smithi012 ceph-mon[31541]: pgmap v605: 129 pgs: 129 active+clean; 847 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.8 MiB/s wr, 744 op/s 2024-03-18T00:04:18.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:17 smithi079 ceph-mon[36241]: pgmap v605: 129 pgs: 129 active+clean; 847 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.8 MiB/s wr, 744 op/s 2024-03-18T00:04:18.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:17 smithi191 ceph-mon[36708]: pgmap v605: 129 pgs: 129 active+clean; 847 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.8 MiB/s wr, 744 op/s 2024-03-18T00:04:19.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:19 smithi012 ceph-mon[31541]: pgmap v606: 129 pgs: 129 active+clean; 830 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.4 MiB/s wr, 664 op/s 2024-03-18T00:04:20.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:19 smithi079 ceph-mon[36241]: pgmap v606: 129 pgs: 129 active+clean; 830 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.4 MiB/s wr, 664 op/s 2024-03-18T00:04:20.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:19 smithi191 ceph-mon[36708]: pgmap v606: 129 pgs: 129 active+clean; 830 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.4 MiB/s wr, 664 op/s 2024-03-18T00:04:21.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:21 smithi012 ceph-mon[31541]: pgmap v607: 129 pgs: 129 active+clean; 818 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 707 op/s 2024-03-18T00:04:22.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:21 smithi079 ceph-mon[36241]: pgmap v607: 129 pgs: 129 active+clean; 818 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 707 op/s 2024-03-18T00:04:22.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:21 smithi191 ceph-mon[36708]: pgmap v607: 129 pgs: 129 active+clean; 818 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 707 op/s 2024-03-18T00:04:23.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:23 smithi012 ceph-mon[31541]: pgmap v608: 129 pgs: 129 active+clean; 816 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 505 op/s 2024-03-18T00:04:23.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:23 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:04:24.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:23 smithi079 ceph-mon[36241]: pgmap v608: 129 pgs: 129 active+clean; 816 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 505 op/s 2024-03-18T00:04:24.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:23 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:04:24.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:23 smithi191 ceph-mon[36708]: pgmap v608: 129 pgs: 129 active+clean; 816 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 505 op/s 2024-03-18T00:04:24.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:23 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:04:25.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:25 smithi012 ceph-mon[31541]: pgmap v609: 129 pgs: 129 active+clean; 808 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 1.3 MiB/s wr, 575 op/s 2024-03-18T00:04:25.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:25 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:04:26.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:25 smithi079 ceph-mon[36241]: pgmap v609: 129 pgs: 129 active+clean; 808 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 1.3 MiB/s wr, 575 op/s 2024-03-18T00:04:26.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:25 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:04:26.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:25 smithi191 ceph-mon[36708]: pgmap v609: 129 pgs: 129 active+clean; 808 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 1.3 MiB/s wr, 575 op/s 2024-03-18T00:04:26.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:25 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:04:27.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:27 smithi012 ceph-mon[31541]: pgmap v610: 129 pgs: 129 active+clean; 800 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 989 KiB/s wr, 618 op/s 2024-03-18T00:04:28.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:27 smithi079 ceph-mon[36241]: pgmap v610: 129 pgs: 129 active+clean; 800 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 989 KiB/s wr, 618 op/s 2024-03-18T00:04:28.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:27 smithi191 ceph-mon[36708]: pgmap v610: 129 pgs: 129 active+clean; 800 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 989 KiB/s wr, 618 op/s 2024-03-18T00:04:29.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:29 smithi012 ceph-mon[31541]: pgmap v611: 129 pgs: 129 active+clean; 796 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 967 KiB/s wr, 593 op/s 2024-03-18T00:04:30.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:29 smithi079 ceph-mon[36241]: pgmap v611: 129 pgs: 129 active+clean; 796 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 967 KiB/s wr, 593 op/s 2024-03-18T00:04:30.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:29 smithi191 ceph-mon[36708]: pgmap v611: 129 pgs: 129 active+clean; 796 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 967 KiB/s wr, 593 op/s 2024-03-18T00:04:31.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:31 smithi012 ceph-mon[31541]: pgmap v612: 129 pgs: 129 active+clean; 786 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 695 op/s 2024-03-18T00:04:32.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:31 smithi079 ceph-mon[36241]: pgmap v612: 129 pgs: 129 active+clean; 786 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 695 op/s 2024-03-18T00:04:32.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:31 smithi191 ceph-mon[36708]: pgmap v612: 129 pgs: 129 active+clean; 786 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 695 op/s 2024-03-18T00:04:34.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:33 smithi079 ceph-mon[36241]: pgmap v613: 129 pgs: 129 active+clean; 784 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 951 KiB/s wr, 567 op/s 2024-03-18T00:04:34.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:33 smithi191 ceph-mon[36708]: pgmap v613: 129 pgs: 129 active+clean; 784 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 951 KiB/s wr, 567 op/s 2024-03-18T00:04:34.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:33 smithi012 ceph-mon[31541]: pgmap v613: 129 pgs: 129 active+clean; 784 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 951 KiB/s wr, 567 op/s 2024-03-18T00:04:36.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:35 smithi079 ceph-mon[36241]: pgmap v614: 129 pgs: 129 active+clean; 780 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 596 op/s 2024-03-18T00:04:36.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:35 smithi191 ceph-mon[36708]: pgmap v614: 129 pgs: 129 active+clean; 780 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 596 op/s 2024-03-18T00:04:36.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:35 smithi012 ceph-mon[31541]: pgmap v614: 129 pgs: 129 active+clean; 780 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 596 op/s 2024-03-18T00:04:37.970 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:37 smithi012 ceph-mon[31541]: pgmap v615: 129 pgs: 129 active+clean; 771 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 592 op/s 2024-03-18T00:04:38.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:37 smithi079 ceph-mon[36241]: pgmap v615: 129 pgs: 129 active+clean; 771 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 592 op/s 2024-03-18T00:04:38.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:37 smithi191 ceph-mon[36708]: pgmap v615: 129 pgs: 129 active+clean; 771 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 592 op/s 2024-03-18T00:04:40.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:39 smithi079 ceph-mon[36241]: pgmap v616: 129 pgs: 129 active+clean; 770 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 532 op/s 2024-03-18T00:04:40.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:39 smithi191 ceph-mon[36708]: pgmap v616: 129 pgs: 129 active+clean; 770 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 532 op/s 2024-03-18T00:04:40.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:39 smithi012 ceph-mon[31541]: pgmap v616: 129 pgs: 129 active+clean; 770 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 532 op/s 2024-03-18T00:04:42.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:41 smithi079 ceph-mon[36241]: pgmap v617: 129 pgs: 129 active+clean; 735 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 603 op/s 2024-03-18T00:04:42.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:41 smithi191 ceph-mon[36708]: pgmap v617: 129 pgs: 129 active+clean; 735 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 603 op/s 2024-03-18T00:04:42.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:41 smithi012 ceph-mon[31541]: pgmap v617: 129 pgs: 129 active+clean; 735 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 603 op/s 2024-03-18T00:04:43.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:42 smithi079 ceph-mon[36241]: pgmap v618: 129 pgs: 129 active+clean; 727 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.0 MiB/s wr, 471 op/s 2024-03-18T00:04:43.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:42 smithi191 ceph-mon[36708]: pgmap v618: 129 pgs: 129 active+clean; 727 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.0 MiB/s wr, 471 op/s 2024-03-18T00:04:43.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:42 smithi012 ceph-mon[31541]: pgmap v618: 129 pgs: 129 active+clean; 727 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.0 MiB/s wr, 471 op/s 2024-03-18T00:04:45.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:45 smithi079 ceph-mon[36241]: pgmap v619: 129 pgs: 129 active+clean; 724 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 471 op/s 2024-03-18T00:04:45.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:45 smithi191 ceph-mon[36708]: pgmap v619: 129 pgs: 129 active+clean; 724 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 471 op/s 2024-03-18T00:04:45.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:45 smithi012 ceph-mon[31541]: pgmap v619: 129 pgs: 129 active+clean; 724 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 471 op/s 2024-03-18T00:04:47.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:47 smithi079 ceph-mon[36241]: pgmap v620: 129 pgs: 129 active+clean; 716 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 468 op/s 2024-03-18T00:04:47.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:47 smithi191 ceph-mon[36708]: pgmap v620: 129 pgs: 129 active+clean; 716 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 468 op/s 2024-03-18T00:04:47.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:47 smithi012 ceph-mon[31541]: pgmap v620: 129 pgs: 129 active+clean; 716 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 468 op/s 2024-03-18T00:04:49.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:49 smithi079 ceph-mon[36241]: pgmap v621: 129 pgs: 129 active+clean; 714 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 410 op/s 2024-03-18T00:04:49.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:49 smithi191 ceph-mon[36708]: pgmap v621: 129 pgs: 129 active+clean; 714 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 410 op/s 2024-03-18T00:04:49.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:49 smithi012 ceph-mon[31541]: pgmap v621: 129 pgs: 129 active+clean; 714 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 410 op/s 2024-03-18T00:04:51.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:51 smithi079 ceph-mon[36241]: pgmap v622: 129 pgs: 129 active+clean; 702 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 506 op/s 2024-03-18T00:04:51.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:51 smithi191 ceph-mon[36708]: pgmap v622: 129 pgs: 129 active+clean; 702 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 506 op/s 2024-03-18T00:04:51.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:51 smithi012 ceph-mon[31541]: pgmap v622: 129 pgs: 129 active+clean; 702 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 506 op/s 2024-03-18T00:04:53.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:53 smithi079 ceph-mon[36241]: pgmap v623: 129 pgs: 129 active+clean; 699 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 420 op/s 2024-03-18T00:04:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:53 smithi191 ceph-mon[36708]: pgmap v623: 129 pgs: 129 active+clean; 699 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 420 op/s 2024-03-18T00:04:53.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:53 smithi012 ceph-mon[31541]: pgmap v623: 129 pgs: 129 active+clean; 699 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 420 op/s 2024-03-18T00:04:55.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:55 smithi079 ceph-mon[36241]: pgmap v624: 129 pgs: 129 active+clean; 692 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 488 op/s 2024-03-18T00:04:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:55 smithi191 ceph-mon[36708]: pgmap v624: 129 pgs: 129 active+clean; 692 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 488 op/s 2024-03-18T00:04:55.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:55 smithi012 ceph-mon[31541]: pgmap v624: 129 pgs: 129 active+clean; 692 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 488 op/s 2024-03-18T00:04:57.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:57 smithi079 ceph-mon[36241]: pgmap v625: 129 pgs: 129 active+clean; 688 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 1013 KiB/s wr, 584 op/s 2024-03-18T00:04:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:57 smithi191 ceph-mon[36708]: pgmap v625: 129 pgs: 129 active+clean; 688 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 1013 KiB/s wr, 584 op/s 2024-03-18T00:04:57.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:57 smithi012 ceph-mon[31541]: pgmap v625: 129 pgs: 129 active+clean; 688 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 1013 KiB/s wr, 584 op/s 2024-03-18T00:04:59.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:04:59 smithi079 ceph-mon[36241]: pgmap v626: 129 pgs: 129 active+clean; 685 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 598 op/s 2024-03-18T00:04:59.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:04:59 smithi191 ceph-mon[36708]: pgmap v626: 129 pgs: 129 active+clean; 685 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 598 op/s 2024-03-18T00:04:59.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:04:59 smithi012 ceph-mon[31541]: pgmap v626: 129 pgs: 129 active+clean; 685 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 598 op/s 2024-03-18T00:05:01.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:01 smithi079 ceph-mon[36241]: pgmap v627: 129 pgs: 129 active+clean; 672 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 940 KiB/s wr, 728 op/s 2024-03-18T00:05:01.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:01 smithi191 ceph-mon[36708]: pgmap v627: 129 pgs: 129 active+clean; 672 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 940 KiB/s wr, 728 op/s 2024-03-18T00:05:01.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:01 smithi012 ceph-mon[31541]: pgmap v627: 129 pgs: 129 active+clean; 672 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 940 KiB/s wr, 728 op/s 2024-03-18T00:05:03.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:03 smithi079 ceph-mon[36241]: pgmap v628: 129 pgs: 129 active+clean; 671 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 829 KiB/s wr, 619 op/s 2024-03-18T00:05:03.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:03 smithi191 ceph-mon[36708]: pgmap v628: 129 pgs: 129 active+clean; 671 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 829 KiB/s wr, 619 op/s 2024-03-18T00:05:03.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:03 smithi012 ceph-mon[31541]: pgmap v628: 129 pgs: 129 active+clean; 671 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 829 KiB/s wr, 619 op/s 2024-03-18T00:05:05.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:05 smithi079 ceph-mon[36241]: pgmap v629: 129 pgs: 129 active+clean; 665 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 971 KiB/s wr, 631 op/s 2024-03-18T00:05:05.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:05 smithi191 ceph-mon[36708]: pgmap v629: 129 pgs: 129 active+clean; 665 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 971 KiB/s wr, 631 op/s 2024-03-18T00:05:05.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:05 smithi012 ceph-mon[31541]: pgmap v629: 129 pgs: 129 active+clean; 665 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 971 KiB/s wr, 631 op/s 2024-03-18T00:05:06.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:06 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:05:06.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:06 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:05:06.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:06 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:05:07.820 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:07 smithi012 ceph-mon[31541]: pgmap v630: 129 pgs: 129 active+clean; 639 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 1.0 MiB/s wr, 615 op/s 2024-03-18T00:05:07.821 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:07 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:05:07.821 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:07 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:05:07.821 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:07 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:05:07.821 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:07 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:05:07.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:07 smithi191 ceph-mon[36708]: pgmap v630: 129 pgs: 129 active+clean; 639 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 1.0 MiB/s wr, 615 op/s 2024-03-18T00:05:07.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:07 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:05:07.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:07 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:05:07.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:07 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:05:07.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:07 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:05:08.154 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:07 smithi079 ceph-mon[36241]: pgmap v630: 129 pgs: 129 active+clean; 639 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 1.0 MiB/s wr, 615 op/s 2024-03-18T00:05:08.154 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:07 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:05:08.154 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:07 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:05:08.154 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:07 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:05:08.154 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:07 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:05:08.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:08 smithi079 ceph-mon[36241]: Adjusting osd_memory_target on smithi191 to 174.1M 2024-03-18T00:05:08.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:08 smithi079 ceph-mon[36241]: Unable to set osd_memory_target on smithi191 to 182639206: error parsing value: Value '182639206' is below minimum 939524096 2024-03-18T00:05:08.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:08 smithi191 ceph-mon[36708]: Adjusting osd_memory_target on smithi191 to 174.1M 2024-03-18T00:05:08.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:08 smithi191 ceph-mon[36708]: Unable to set osd_memory_target on smithi191 to 182639206: error parsing value: Value '182639206' is below minimum 939524096 2024-03-18T00:05:08.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:08 smithi012 ceph-mon[31541]: Adjusting osd_memory_target on smithi191 to 174.1M 2024-03-18T00:05:08.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:08 smithi012 ceph-mon[31541]: Unable to set osd_memory_target on smithi191 to 182639206: error parsing value: Value '182639206' is below minimum 939524096 2024-03-18T00:05:09.815 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:09 smithi079 ceph-mon[36241]: pgmap v631: 129 pgs: 129 active+clean; 627 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 833 KiB/s wr, 538 op/s 2024-03-18T00:05:09.815 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:09 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:09 smithi191 ceph-mon[36708]: pgmap v631: 129 pgs: 129 active+clean; 627 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 833 KiB/s wr, 538 op/s 2024-03-18T00:05:09.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:09 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:09.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:09 smithi012 ceph-mon[31541]: pgmap v631: 129 pgs: 129 active+clean; 627 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 833 KiB/s wr, 538 op/s 2024-03-18T00:05:09.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:09 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:10.801 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:10 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:10.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:10 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:10.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:10 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:11.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:11 smithi191 ceph-mon[36708]: pgmap v632: 129 pgs: 129 active+clean; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1003 KiB/s wr, 617 op/s 2024-03-18T00:05:11.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:05:11.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:05:11.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:05:11.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:11 smithi012 ceph-mon[31541]: pgmap v632: 129 pgs: 129 active+clean; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1003 KiB/s wr, 617 op/s 2024-03-18T00:05:11.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:05:11.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:05:11.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:05:12.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:11 smithi079 ceph-mon[36241]: pgmap v632: 129 pgs: 129 active+clean; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1003 KiB/s wr, 617 op/s 2024-03-18T00:05:12.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:05:12.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:05:12.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:05:13.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:13 smithi191 ceph-mon[36708]: pgmap v633: 129 pgs: 129 active+clean; 604 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 948 KiB/s wr, 473 op/s 2024-03-18T00:05:13.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:13 smithi012 ceph-mon[31541]: pgmap v633: 129 pgs: 129 active+clean; 604 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 948 KiB/s wr, 473 op/s 2024-03-18T00:05:14.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:13 smithi079 ceph-mon[36241]: pgmap v633: 129 pgs: 129 active+clean; 604 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 948 KiB/s wr, 473 op/s 2024-03-18T00:05:15.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:15 smithi191 ceph-mon[36708]: pgmap v634: 129 pgs: 129 active+clean; 597 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 883 KiB/s wr, 482 op/s 2024-03-18T00:05:15.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:15 smithi012 ceph-mon[31541]: pgmap v634: 129 pgs: 129 active+clean; 597 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 883 KiB/s wr, 482 op/s 2024-03-18T00:05:16.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:15 smithi079 ceph-mon[36241]: pgmap v634: 129 pgs: 129 active+clean; 597 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 883 KiB/s wr, 482 op/s 2024-03-18T00:05:17.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:17 smithi191 ceph-mon[36708]: pgmap v635: 129 pgs: 129 active+clean; 597 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 486 op/s 2024-03-18T00:05:17.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:17 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:17.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:17 smithi012 ceph-mon[31541]: pgmap v635: 129 pgs: 129 active+clean; 597 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 486 op/s 2024-03-18T00:05:17.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:17 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:18.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:17 smithi079 ceph-mon[36241]: pgmap v635: 129 pgs: 129 active+clean; 597 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 486 op/s 2024-03-18T00:05:18.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:17 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:19.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:19 smithi191 ceph-mon[36708]: pgmap v636: 129 pgs: 129 active+clean; 603 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 441 op/s 2024-03-18T00:05:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:19 smithi012 ceph-mon[31541]: pgmap v636: 129 pgs: 129 active+clean; 603 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 441 op/s 2024-03-18T00:05:20.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:19 smithi079 ceph-mon[36241]: pgmap v636: 129 pgs: 129 active+clean; 603 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 441 op/s 2024-03-18T00:05:21.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:21 smithi191 ceph-mon[36708]: pgmap v637: 129 pgs: 129 active+clean; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 490 op/s 2024-03-18T00:05:21.847 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:21 smithi079 ceph-mon[36241]: pgmap v637: 129 pgs: 129 active+clean; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 490 op/s 2024-03-18T00:05:21.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:21 smithi012 ceph-mon[31541]: pgmap v637: 129 pgs: 129 active+clean; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 490 op/s 2024-03-18T00:05:23.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:23 smithi191 ceph-mon[36708]: pgmap v638: 129 pgs: 129 active+clean; 605 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 366 op/s 2024-03-18T00:05:23.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:23 smithi012 ceph-mon[31541]: pgmap v638: 129 pgs: 129 active+clean; 605 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 366 op/s 2024-03-18T00:05:24.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:23 smithi079 ceph-mon[36241]: pgmap v638: 129 pgs: 129 active+clean; 605 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 366 op/s 2024-03-18T00:05:25.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:25 smithi012 ceph-mon[31541]: pgmap v639: 129 pgs: 129 active+clean; 603 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 407 op/s 2024-03-18T00:05:26.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:25 smithi079 ceph-mon[36241]: pgmap v639: 129 pgs: 129 active+clean; 603 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 407 op/s 2024-03-18T00:05:26.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:25 smithi191 ceph-mon[36708]: pgmap v639: 129 pgs: 129 active+clean; 603 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 407 op/s 2024-03-18T00:05:26.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:26 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:27.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:26 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:27.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:26 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:27.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:27 smithi012 ceph-mon[31541]: pgmap v640: 129 pgs: 129 active+clean; 588 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 452 op/s 2024-03-18T00:05:28.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:27 smithi079 ceph-mon[36241]: pgmap v640: 129 pgs: 129 active+clean; 588 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 452 op/s 2024-03-18T00:05:28.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:27 smithi191 ceph-mon[36708]: pgmap v640: 129 pgs: 129 active+clean; 588 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 452 op/s 2024-03-18T00:05:29.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:29 smithi012 ceph-mon[31541]: pgmap v641: 129 pgs: 129 active+clean; 587 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 444 op/s 2024-03-18T00:05:30.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:29 smithi079 ceph-mon[36241]: pgmap v641: 129 pgs: 129 active+clean; 587 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 444 op/s 2024-03-18T00:05:30.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:29 smithi191 ceph-mon[36708]: pgmap v641: 129 pgs: 129 active+clean; 587 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 444 op/s 2024-03-18T00:05:31.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:31 smithi012 ceph-mon[31541]: pgmap v642: 129 pgs: 129 active+clean; 573 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 602 op/s 2024-03-18T00:05:32.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:31 smithi079 ceph-mon[36241]: pgmap v642: 129 pgs: 129 active+clean; 573 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 602 op/s 2024-03-18T00:05:32.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:31 smithi191 ceph-mon[36708]: pgmap v642: 129 pgs: 129 active+clean; 573 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 602 op/s 2024-03-18T00:05:33.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:33 smithi012 ceph-mon[31541]: pgmap v643: 129 pgs: 129 active+clean; 569 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 538 op/s 2024-03-18T00:05:34.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:33 smithi079 ceph-mon[36241]: pgmap v643: 129 pgs: 129 active+clean; 569 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 538 op/s 2024-03-18T00:05:34.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:33 smithi191 ceph-mon[36708]: pgmap v643: 129 pgs: 129 active+clean; 569 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 538 op/s 2024-03-18T00:05:35.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:35 smithi012 ceph-mon[31541]: pgmap v644: 129 pgs: 129 active+clean; 563 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 586 op/s 2024-03-18T00:05:36.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:35 smithi079 ceph-mon[36241]: pgmap v644: 129 pgs: 129 active+clean; 563 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 586 op/s 2024-03-18T00:05:36.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:35 smithi191 ceph-mon[36708]: pgmap v644: 129 pgs: 129 active+clean; 563 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 586 op/s 2024-03-18T00:05:37.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:37 smithi012 ceph-mon[31541]: pgmap v645: 129 pgs: 129 active+clean; 553 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 606 op/s 2024-03-18T00:05:37.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:37 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:38.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:37 smithi079 ceph-mon[36241]: pgmap v645: 129 pgs: 129 active+clean; 553 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 606 op/s 2024-03-18T00:05:38.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:37 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:38.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:37 smithi191 ceph-mon[36708]: pgmap v645: 129 pgs: 129 active+clean; 553 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 606 op/s 2024-03-18T00:05:38.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:37 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:39.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:39 smithi012 ceph-mon[31541]: pgmap v646: 129 pgs: 129 active+clean; 540 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 1.1 MiB/s wr, 553 op/s 2024-03-18T00:05:40.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:39 smithi079 ceph-mon[36241]: pgmap v646: 129 pgs: 129 active+clean; 540 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 1.1 MiB/s wr, 553 op/s 2024-03-18T00:05:40.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:39 smithi191 ceph-mon[36708]: pgmap v646: 129 pgs: 129 active+clean; 540 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 1.1 MiB/s wr, 553 op/s 2024-03-18T00:05:42.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:41 smithi079 ceph-mon[36241]: pgmap v647: 129 pgs: 129 active+clean; 511 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 618 op/s 2024-03-18T00:05:42.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:41 smithi191 ceph-mon[36708]: pgmap v647: 129 pgs: 129 active+clean; 511 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 618 op/s 2024-03-18T00:05:42.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:41 smithi012 ceph-mon[31541]: pgmap v647: 129 pgs: 129 active+clean; 511 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 618 op/s 2024-03-18T00:05:44.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:43 smithi079 ceph-mon[36241]: pgmap v648: 129 pgs: 129 active+clean; 505 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 462 op/s 2024-03-18T00:05:44.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:43 smithi191 ceph-mon[36708]: pgmap v648: 129 pgs: 129 active+clean; 505 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 462 op/s 2024-03-18T00:05:44.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:43 smithi012 ceph-mon[31541]: pgmap v648: 129 pgs: 129 active+clean; 505 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 462 op/s 2024-03-18T00:05:46.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:45 smithi079 ceph-mon[36241]: pgmap v649: 129 pgs: 129 active+clean; 496 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 515 op/s 2024-03-18T00:05:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:45 smithi191 ceph-mon[36708]: pgmap v649: 129 pgs: 129 active+clean; 496 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 515 op/s 2024-03-18T00:05:46.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:45 smithi012 ceph-mon[31541]: pgmap v649: 129 pgs: 129 active+clean; 496 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 515 op/s 2024-03-18T00:05:48.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:47 smithi079 ceph-mon[36241]: pgmap v650: 129 pgs: 129 active+clean; 490 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 559 op/s 2024-03-18T00:05:48.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:47 smithi191 ceph-mon[36708]: pgmap v650: 129 pgs: 129 active+clean; 490 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 559 op/s 2024-03-18T00:05:48.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:47 smithi012 ceph-mon[31541]: pgmap v650: 129 pgs: 129 active+clean; 490 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 559 op/s 2024-03-18T00:05:50.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:49 smithi079 ceph-mon[36241]: pgmap v651: 129 pgs: 129 active+clean; 483 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 1.0 MiB/s wr, 550 op/s 2024-03-18T00:05:50.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:49 smithi191 ceph-mon[36708]: pgmap v651: 129 pgs: 129 active+clean; 483 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 1.0 MiB/s wr, 550 op/s 2024-03-18T00:05:50.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:49 smithi012 ceph-mon[31541]: pgmap v651: 129 pgs: 129 active+clean; 483 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 1.0 MiB/s wr, 550 op/s 2024-03-18T00:05:52.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:51 smithi079 ceph-mon[36241]: pgmap v652: 129 pgs: 129 active+clean; 473 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 678 op/s 2024-03-18T00:05:52.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:51 smithi191 ceph-mon[36708]: pgmap v652: 129 pgs: 129 active+clean; 473 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 678 op/s 2024-03-18T00:05:52.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:51 smithi012 ceph-mon[31541]: pgmap v652: 129 pgs: 129 active+clean; 473 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 678 op/s 2024-03-18T00:05:53.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:52 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:53.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:52 smithi079 ceph-mon[36241]: pgmap v653: 129 pgs: 129 active+clean; 469 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 1.0 MiB/s wr, 600 op/s 2024-03-18T00:05:53.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:52 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:53.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:52 smithi191 ceph-mon[36708]: pgmap v653: 129 pgs: 129 active+clean; 469 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 1.0 MiB/s wr, 600 op/s 2024-03-18T00:05:53.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:52 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:53.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:52 smithi012 ceph-mon[31541]: pgmap v653: 129 pgs: 129 active+clean; 469 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 1.0 MiB/s wr, 600 op/s 2024-03-18T00:05:55.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:55 smithi079 ceph-mon[36241]: pgmap v654: 129 pgs: 129 active+clean; 466 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 617 op/s 2024-03-18T00:05:55.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:55 smithi191 ceph-mon[36708]: pgmap v654: 129 pgs: 129 active+clean; 466 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 617 op/s 2024-03-18T00:05:55.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:55 smithi012 ceph-mon[31541]: pgmap v654: 129 pgs: 129 active+clean; 466 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 617 op/s 2024-03-18T00:05:56.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:56 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:56.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:56 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:56.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:56 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:05:57.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:57 smithi079 ceph-mon[36241]: pgmap v655: 129 pgs: 129 active+clean; 455 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 609 op/s 2024-03-18T00:05:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:57 smithi191 ceph-mon[36708]: pgmap v655: 129 pgs: 129 active+clean; 455 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 609 op/s 2024-03-18T00:05:57.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:57 smithi012 ceph-mon[31541]: pgmap v655: 129 pgs: 129 active+clean; 455 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 609 op/s 2024-03-18T00:06:00.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:05:59 smithi079 ceph-mon[36241]: pgmap v656: 129 pgs: 129 active+clean; 453 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 544 op/s 2024-03-18T00:06:00.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:05:59 smithi191 ceph-mon[36708]: pgmap v656: 129 pgs: 129 active+clean; 453 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 544 op/s 2024-03-18T00:06:00.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:05:59 smithi012 ceph-mon[31541]: pgmap v656: 129 pgs: 129 active+clean; 453 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 544 op/s 2024-03-18T00:06:02.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:01 smithi079 ceph-mon[36241]: pgmap v657: 129 pgs: 129 active+clean; 418 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 622 op/s 2024-03-18T00:06:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:01 smithi191 ceph-mon[36708]: pgmap v657: 129 pgs: 129 active+clean; 418 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 622 op/s 2024-03-18T00:06:02.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:01 smithi012 ceph-mon[31541]: pgmap v657: 129 pgs: 129 active+clean; 418 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 622 op/s 2024-03-18T00:06:04.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:03 smithi079 ceph-mon[36241]: pgmap v658: 129 pgs: 129 active+clean; 407 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 955 KiB/s wr, 482 op/s 2024-03-18T00:06:04.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:03 smithi191 ceph-mon[36708]: pgmap v658: 129 pgs: 129 active+clean; 407 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 955 KiB/s wr, 482 op/s 2024-03-18T00:06:04.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:03 smithi012 ceph-mon[31541]: pgmap v658: 129 pgs: 129 active+clean; 407 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 955 KiB/s wr, 482 op/s 2024-03-18T00:06:06.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:05 smithi079 ceph-mon[36241]: pgmap v659: 129 pgs: 129 active+clean; 405 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 500 op/s 2024-03-18T00:06:06.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:05 smithi191 ceph-mon[36708]: pgmap v659: 129 pgs: 129 active+clean; 405 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 500 op/s 2024-03-18T00:06:06.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:05 smithi012 ceph-mon[31541]: pgmap v659: 129 pgs: 129 active+clean; 405 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 500 op/s 2024-03-18T00:06:07.970 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:07 smithi012 ceph-mon[31541]: pgmap v660: 129 pgs: 129 active+clean; 395 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 543 op/s 2024-03-18T00:06:08.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:07 smithi079 ceph-mon[36241]: pgmap v660: 129 pgs: 129 active+clean; 395 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 543 op/s 2024-03-18T00:06:08.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:07 smithi191 ceph-mon[36708]: pgmap v660: 129 pgs: 129 active+clean; 395 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 543 op/s 2024-03-18T00:06:10.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:09 smithi079 ceph-mon[36241]: pgmap v661: 129 pgs: 129 active+clean; 389 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 1.0 MiB/s wr, 525 op/s 2024-03-18T00:06:10.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:09 smithi191 ceph-mon[36708]: pgmap v661: 129 pgs: 129 active+clean; 389 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 1.0 MiB/s wr, 525 op/s 2024-03-18T00:06:10.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:09 smithi012 ceph-mon[31541]: pgmap v661: 129 pgs: 129 active+clean; 389 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 1.0 MiB/s wr, 525 op/s 2024-03-18T00:06:12.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:11 smithi079 ceph-mon[36241]: pgmap v662: 129 pgs: 129 active+clean; 379 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 615 op/s 2024-03-18T00:06:12.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:11 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:06:12.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:11 smithi191 ceph-mon[36708]: pgmap v662: 129 pgs: 129 active+clean; 379 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 615 op/s 2024-03-18T00:06:12.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:11 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:06:12.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:11 smithi012 ceph-mon[31541]: pgmap v662: 129 pgs: 129 active+clean; 379 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 615 op/s 2024-03-18T00:06:12.172 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:11 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:06:13.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:06:13.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:06:13.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:12 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:06:13.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:06:13.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:06:13.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:12 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:06:13.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:06:13.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:06:13.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:12 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:06:14.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:13 smithi079 ceph-mon[36241]: pgmap v663: 129 pgs: 129 active+clean; 377 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 1001 KiB/s wr, 492 op/s 2024-03-18T00:06:14.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:13 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:06:14.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:13 smithi191 ceph-mon[36708]: pgmap v663: 129 pgs: 129 active+clean; 377 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 1001 KiB/s wr, 492 op/s 2024-03-18T00:06:14.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:13 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:06:14.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:13 smithi012 ceph-mon[31541]: pgmap v663: 129 pgs: 129 active+clean; 377 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 1001 KiB/s wr, 492 op/s 2024-03-18T00:06:14.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:13 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:06:15.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:14 smithi079 ceph-mon[36241]: pgmap v664: 129 pgs: 129 active+clean; 370 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 1.0 MiB/s wr, 529 op/s 2024-03-18T00:06:15.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:14 smithi191 ceph-mon[36708]: pgmap v664: 129 pgs: 129 active+clean; 370 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 1.0 MiB/s wr, 529 op/s 2024-03-18T00:06:15.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:14 smithi012 ceph-mon[31541]: pgmap v664: 129 pgs: 129 active+clean; 370 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 1.0 MiB/s wr, 529 op/s 2024-03-18T00:06:17.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:17 smithi079 ceph-mon[36241]: pgmap v665: 129 pgs: 129 active+clean; 365 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 1.0 MiB/s wr, 552 op/s 2024-03-18T00:06:17.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:17 smithi191 ceph-mon[36708]: pgmap v665: 129 pgs: 129 active+clean; 365 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 1.0 MiB/s wr, 552 op/s 2024-03-18T00:06:17.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:17 smithi012 ceph-mon[31541]: pgmap v665: 129 pgs: 129 active+clean; 365 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 1.0 MiB/s wr, 552 op/s 2024-03-18T00:06:19.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:19 smithi079 ceph-mon[36241]: pgmap v666: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 960 KiB/s wr, 506 op/s 2024-03-18T00:06:19.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:19 smithi191 ceph-mon[36708]: pgmap v666: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 960 KiB/s wr, 506 op/s 2024-03-18T00:06:19.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:19 smithi012 ceph-mon[31541]: pgmap v666: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 960 KiB/s wr, 506 op/s 2024-03-18T00:06:21.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:21 smithi079 ceph-mon[36241]: pgmap v667: 129 pgs: 129 active+clean; 329 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 557 op/s 2024-03-18T00:06:21.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:21 smithi191 ceph-mon[36708]: pgmap v667: 129 pgs: 129 active+clean; 329 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 557 op/s 2024-03-18T00:06:21.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:21 smithi012 ceph-mon[31541]: pgmap v667: 129 pgs: 129 active+clean; 329 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 557 op/s 2024-03-18T00:06:23.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:23 smithi079 ceph-mon[36241]: pgmap v668: 129 pgs: 129 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 962 KiB/s wr, 458 op/s 2024-03-18T00:06:23.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:23 smithi191 ceph-mon[36708]: pgmap v668: 129 pgs: 129 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 962 KiB/s wr, 458 op/s 2024-03-18T00:06:23.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:23 smithi012 ceph-mon[31541]: pgmap v668: 129 pgs: 129 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 962 KiB/s wr, 458 op/s 2024-03-18T00:06:25.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:25 smithi079 ceph-mon[36241]: pgmap v669: 129 pgs: 129 active+clean; 318 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 510 op/s 2024-03-18T00:06:25.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:25 smithi191 ceph-mon[36708]: pgmap v669: 129 pgs: 129 active+clean; 318 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 510 op/s 2024-03-18T00:06:25.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:25 smithi012 ceph-mon[31541]: pgmap v669: 129 pgs: 129 active+clean; 318 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 510 op/s 2024-03-18T00:06:27.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:27 smithi079 ceph-mon[36241]: pgmap v670: 129 pgs: 129 active+clean; 308 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 544 op/s 2024-03-18T00:06:27.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:27 smithi191 ceph-mon[36708]: pgmap v670: 129 pgs: 129 active+clean; 308 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 544 op/s 2024-03-18T00:06:27.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:27 smithi012 ceph-mon[31541]: pgmap v670: 129 pgs: 129 active+clean; 308 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 544 op/s 2024-03-18T00:06:29.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:29 smithi079 ceph-mon[36241]: pgmap v671: 129 pgs: 129 active+clean; 302 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 949 KiB/s wr, 504 op/s 2024-03-18T00:06:29.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:29 smithi191 ceph-mon[36708]: pgmap v671: 129 pgs: 129 active+clean; 302 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 949 KiB/s wr, 504 op/s 2024-03-18T00:06:29.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:29 smithi012 ceph-mon[31541]: pgmap v671: 129 pgs: 129 active+clean; 302 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 949 KiB/s wr, 504 op/s 2024-03-18T00:06:31.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:31 smithi079 ceph-mon[36241]: pgmap v672: 129 pgs: 129 active+clean; 291 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 582 op/s 2024-03-18T00:06:31.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:31 smithi191 ceph-mon[36708]: pgmap v672: 129 pgs: 129 active+clean; 291 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 582 op/s 2024-03-18T00:06:31.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:31 smithi012 ceph-mon[31541]: pgmap v672: 129 pgs: 129 active+clean; 291 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 582 op/s 2024-03-18T00:06:33.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:33 smithi079 ceph-mon[36241]: pgmap v673: 129 pgs: 129 active+clean; 288 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 503 op/s 2024-03-18T00:06:33.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:33 smithi191 ceph-mon[36708]: pgmap v673: 129 pgs: 129 active+clean; 288 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 503 op/s 2024-03-18T00:06:33.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:33 smithi012 ceph-mon[31541]: pgmap v673: 129 pgs: 129 active+clean; 288 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 503 op/s 2024-03-18T00:06:35.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:35 smithi079 ceph-mon[36241]: pgmap v674: 129 pgs: 129 active+clean; 268 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 529 op/s 2024-03-18T00:06:35.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:35 smithi191 ceph-mon[36708]: pgmap v674: 129 pgs: 129 active+clean; 268 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 529 op/s 2024-03-18T00:06:35.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:35 smithi012 ceph-mon[31541]: pgmap v674: 129 pgs: 129 active+clean; 268 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 529 op/s 2024-03-18T00:06:37.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:37 smithi079 ceph-mon[36241]: pgmap v675: 129 pgs: 129 active+clean; 253 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 532 op/s 2024-03-18T00:06:37.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:37 smithi191 ceph-mon[36708]: pgmap v675: 129 pgs: 129 active+clean; 253 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 532 op/s 2024-03-18T00:06:37.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:37 smithi012 ceph-mon[31541]: pgmap v675: 129 pgs: 129 active+clean; 253 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 532 op/s 2024-03-18T00:06:39.366 INFO:tasks.workunit:Stopping ['suites/blogbench.sh'] on client.0... 2024-03-18T00:06:39.366 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-18T00:06:39.668 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:39 smithi012 ceph-mon[31541]: pgmap v676: 129 pgs: 129 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 483 op/s 2024-03-18T00:06:39.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:39 smithi079 ceph-mon[36241]: pgmap v676: 129 pgs: 129 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 483 op/s 2024-03-18T00:06:39.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:39 smithi191 ceph-mon[36708]: pgmap v676: 129 pgs: 129 active+clean; 248 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 483 op/s 2024-03-18T00:06:40.604 DEBUG:teuthology.parallel:result is None 2024-03-18T00:06:40.605 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T00:06:40.700 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T00:06:40.700 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-18T00:06:40.745 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:06:40.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.951+0000 7f996f200640 1 -- 172.21.15.12:0/4044244437 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f99681053d0 msgr2=0x7f996810bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:40.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.951+0000 7f996f200640 1 --2- 172.21.15.12:0/4044244437 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f99681053d0 0x7f996810bc60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9964009ca0 tx=0x7f996402c100 comp rx=0 tx=0).stop 2024-03-18T00:06:40.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.952+0000 7f996f200640 1 -- 172.21.15.12:0/4044244437 shutdown_connections 2024-03-18T00:06:40.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.952+0000 7f996f200640 1 --2- 172.21.15.12:0/4044244437 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f99681053d0 0x7f996810bc60 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:40.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.952+0000 7f996f200640 1 --2- 172.21.15.12:0/4044244437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9968104a10 0x7f9968104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:40.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.952+0000 7f996f200640 1 --2- 172.21.15.12:0/4044244437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9968103820 0x7f9968103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:40.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.952+0000 7f996f200640 1 -- 172.21.15.12:0/4044244437 >> 172.21.15.12:0/4044244437 conn(0x7f99680fedb0 msgr2=0x7f99681011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:06:40.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.952+0000 7f996f200640 1 -- 172.21.15.12:0/4044244437 shutdown_connections 2024-03-18T00:06:40.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.952+0000 7f996f200640 1 -- 172.21.15.12:0/4044244437 wait complete. 2024-03-18T00:06:40.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.953+0000 7f996f200640 1 Processor -- start 2024-03-18T00:06:40.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.953+0000 7f996f200640 1 -- start start 2024-03-18T00:06:40.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.953+0000 7f996f200640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9968103820 0x7f9968199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:40.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.953+0000 7f996f200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9968104a10 0x7f996819a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:40.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.953+0000 7f996f200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99681053d0 0x7f99681a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:40.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.953+0000 7f996f200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f996810f480 con 0x7f99681053d0 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.953+0000 7f996f200640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f996810f300 con 0x7f9968103820 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.953+0000 7f996f200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f996810f600 con 0x7f9968104a10 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.953+0000 7f996de00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9968103820 0x7f9968199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.953+0000 7f996d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9968104a10 0x7f996819a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.953+0000 7f996de00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9968103820 0x7f9968199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:41756/0 (socket says 172.21.15.12:41756) 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.953+0000 7f996de00640 1 -- 172.21.15.12:0/1166507136 learned_addr learned my addr 172.21.15.12:0/1166507136 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.953+0000 7f996e800640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99681053d0 0x7f99681a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.954+0000 7f996de00640 1 -- 172.21.15.12:0/1166507136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9968104a10 msgr2=0x7f996819a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.954+0000 7f996de00640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9968104a10 0x7f996819a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.954+0000 7f996de00640 1 -- 172.21.15.12:0/1166507136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99681053d0 msgr2=0x7f99681a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.954+0000 7f996de00640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99681053d0 0x7f99681a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.954+0000 7f996de00640 1 -- 172.21.15.12:0/1166507136 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99681a1700 con 0x7f9968103820 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.954+0000 7f996d400640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9968104a10 0x7f996819a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.954+0000 7f996de00640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9968103820 0x7f9968199ae0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f995c00e9c0 tx=0x7f995c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:06:40.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.954+0000 7f9962a00640 1 -- 172.21.15.12:0/1166507136 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c00cd80 con 0x7f9968103820 2024-03-18T00:06:40.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.954+0000 7f9962a00640 1 -- 172.21.15.12:0/1166507136 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f995c0035f0 con 0x7f9968103820 2024-03-18T00:06:40.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.954+0000 7f9962a00640 1 -- 172.21.15.12:0/1166507136 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c010690 con 0x7f9968103820 2024-03-18T00:06:40.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.954+0000 7f996f200640 1 -- 172.21.15.12:0/1166507136 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99681a19f0 con 0x7f9968103820 2024-03-18T00:06:40.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.954+0000 7f996f200640 1 -- 172.21.15.12:0/1166507136 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99681a1ef0 con 0x7f9968103820 2024-03-18T00:06:40.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.955+0000 7f996f200640 1 -- 172.21.15.12:0/1166507136 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99680684f0 con 0x7f9968103820 2024-03-18T00:06:40.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.958+0000 7f9962a00640 1 -- 172.21.15.12:0/1166507136 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f995c0037a0 con 0x7f9968103820 2024-03-18T00:06:40.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.959+0000 7f9962a00640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f99500a2880 0x7f99500a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:40.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.959+0000 7f9962a00640 1 -- 172.21.15.12:0/1166507136 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f995c0cf200 con 0x7f9968103820 2024-03-18T00:06:40.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.959+0000 7f996d400640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f99500a2880 0x7f99500a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:40.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.960+0000 7f996d400640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f99500a2880 0x7f99500a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f996819aff0 tx=0x7f9958005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:06:40.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:40.961+0000 7f9962a00640 1 -- 172.21.15.12:0/1166507136 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f995c09bf50 con 0x7f9968103820 2024-03-18T00:06:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.175+0000 7f996f200640 1 -- 172.21.15.12:0/1166507136 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f996802d240 con 0x7f9968103820 2024-03-18T00:06:41.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.176+0000 7f9962a00640 1 -- 172.21.15.12:0/1166507136 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24618 (secure 0 0 0) 0x7f995c0ab040 con 0x7f9968103820 2024-03-18T00:06:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-18T00:06:41.179 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":94,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:56:55.719277+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:06:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.179+0000 7f996f200640 1 -- 172.21.15.12:0/1166507136 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f99500a2880 msgr2=0x7f99500a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.179+0000 7f996f200640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f99500a2880 0x7f99500a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f996819aff0 tx=0x7f9958005f90 comp rx=0 tx=0).stop 2024-03-18T00:06:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.179+0000 7f996f200640 1 -- 172.21.15.12:0/1166507136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9968103820 msgr2=0x7f9968199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.179+0000 7f996f200640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9968103820 0x7f9968199ae0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f995c00e9c0 tx=0x7f995c00ee80 comp rx=0 tx=0).stop 2024-03-18T00:06:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.179+0000 7f996f200640 1 -- 172.21.15.12:0/1166507136 shutdown_connections 2024-03-18T00:06:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.179+0000 7f996f200640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f99500a2880 0x7f99500a4d40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.179+0000 7f996f200640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99681053d0 0x7f99681a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.179+0000 7f996f200640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9968104a10 0x7f996819a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.179+0000 7f996f200640 1 --2- 172.21.15.12:0/1166507136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9968103820 0x7f9968199ae0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.179+0000 7f996f200640 1 -- 172.21.15.12:0/1166507136 >> 172.21.15.12:0/1166507136 conn(0x7f99680fedb0 msgr2=0x7f9968107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:06:41.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.180+0000 7f996f200640 1 -- 172.21.15.12:0/1166507136 shutdown_connections 2024-03-18T00:06:41.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.180+0000 7f996f200640 1 -- 172.21.15.12:0/1166507136 wait complete. 2024-03-18T00:06:41.200 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:49:38.640282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:54:54.540345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:54:55.803926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:55:00.355066+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T00:06:41.200 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T00:06:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.405+0000 7f3a1f800640 1 -- 172.21.15.12:0/3484169797 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a181096a0 msgr2=0x7f3a1810bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.405+0000 7f3a1f800640 1 --2- 172.21.15.12:0/3484169797 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a181096a0 0x7f3a1810bb30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009ca0 tx=0x7f3a1002c180 comp rx=0 tx=0).stop 2024-03-18T00:06:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.405+0000 7f3a1f800640 1 -- 172.21.15.12:0/3484169797 shutdown_connections 2024-03-18T00:06:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.405+0000 7f3a1f800640 1 --2- 172.21.15.12:0/3484169797 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a181096a0 0x7f3a1810bb30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.405+0000 7f3a1f800640 1 --2- 172.21.15.12:0/3484169797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a180741c0 0x7f3a18074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.405+0000 7f3a1f800640 1 --2- 172.21.15.12:0/3484169797 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a180757a0 0x7f3a18073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.405+0000 7f3a1f800640 1 -- 172.21.15.12:0/3484169797 >> 172.21.15.12:0/3484169797 conn(0x7f3a180fcd80 msgr2=0x7f3a180ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:06:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.405+0000 7f3a1f800640 1 -- 172.21.15.12:0/3484169797 shutdown_connections 2024-03-18T00:06:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.405+0000 7f3a1f800640 1 -- 172.21.15.12:0/3484169797 wait complete. 2024-03-18T00:06:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.406+0000 7f3a1f800640 1 Processor -- start 2024-03-18T00:06:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.406+0000 7f3a1f800640 1 -- start start 2024-03-18T00:06:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1f800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a180741c0 0x7f3a18199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1f800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a180757a0 0x7f3a1819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1f800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a181096a0 0x7f3a181a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1f800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1810f660 con 0x7f3a180757a0 2024-03-18T00:06:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1f800640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1810f4e0 con 0x7f3a181096a0 2024-03-18T00:06:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1f800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1810f7e0 con 0x7f3a180741c0 2024-03-18T00:06:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1ce00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a180741c0 0x7f3a18199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1ce00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a180741c0 0x7f3a18199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:35084/0 (socket says 172.21.15.12:35084) 2024-03-18T00:06:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1ce00640 1 -- 172.21.15.12:0/1000542715 learned_addr learned my addr 172.21.15.12:0/1000542715 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:06:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1d800640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a181096a0 0x7f3a181a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a17e00640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a180757a0 0x7f3a1819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1ce00640 1 -- 172.21.15.12:0/1000542715 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a181096a0 msgr2=0x7f3a181a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1ce00640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a181096a0 0x7f3a181a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1ce00640 1 -- 172.21.15.12:0/1000542715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a180757a0 msgr2=0x7f3a1819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1ce00640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a180757a0 0x7f3a1819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a1ce00640 1 -- 172.21.15.12:0/1000542715 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a181a1900 con 0x7f3a180741c0 2024-03-18T00:06:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.407+0000 7f3a17e00640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a180757a0 0x7f3a1819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:06:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.408+0000 7f3a1ce00640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a180741c0 0x7f3a18199cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00e9f0 tx=0x7f3a0c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:06:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.408+0000 7f3a15600640 1 -- 172.21.15.12:0/1000542715 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c00cd80 con 0x7f3a180741c0 2024-03-18T00:06:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.408+0000 7f3a15600640 1 -- 172.21.15.12:0/1000542715 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a0c010910 con 0x7f3a180741c0 2024-03-18T00:06:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.408+0000 7f3a1f800640 1 -- 172.21.15.12:0/1000542715 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a181a1bf0 con 0x7f3a180741c0 2024-03-18T00:06:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.408+0000 7f3a15600640 1 -- 172.21.15.12:0/1000542715 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c018990 con 0x7f3a180741c0 2024-03-18T00:06:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.408+0000 7f3a1f800640 1 -- 172.21.15.12:0/1000542715 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a181a2190 con 0x7f3a180741c0 2024-03-18T00:06:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.409+0000 7f3a1f800640 1 -- 172.21.15.12:0/1000542715 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a18108460 con 0x7f3a180741c0 2024-03-18T00:06:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.411+0000 7f3a15600640 1 -- 172.21.15.12:0/1000542715 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f3a0c010ac0 con 0x7f3a180741c0 2024-03-18T00:06:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.414+0000 7f3a15600640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3a000a2880 0x7f3a000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.414+0000 7f3a15600640 1 -- 172.21.15.12:0/1000542715 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f3a0c0cf1a0 con 0x7f3a180741c0 2024-03-18T00:06:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.414+0000 7f3a17e00640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3a000a2880 0x7f3a000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.415+0000 7f3a17e00640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3a000a2880 0x7f3a000a4d40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3a040058c0 tx=0x7f3a04005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:06:41.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.417+0000 7f3a15600640 1 -- 172.21.15.12:0/1000542715 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3a0c09be60 con 0x7f3a180741c0 2024-03-18T00:06:41.700 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-18T00:06:41.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.696+0000 7f3a1f800640 1 -- 172.21.15.12:0/1000542715 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3a180623e0 con 0x7f3a180741c0 2024-03-18T00:06:41.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.697+0000 7f3a15600640 1 -- 172.21.15.12:0/1000542715 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f3a0c09b760 con 0x7f3a180741c0 2024-03-18T00:06:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.702+0000 7f3a1f800640 1 -- 172.21.15.12:0/1000542715 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3a000a2880 msgr2=0x7f3a000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.702+0000 7f3a1f800640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3a000a2880 0x7f3a000a4d40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3a040058c0 tx=0x7f3a04005850 comp rx=0 tx=0).stop 2024-03-18T00:06:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.702+0000 7f3a1f800640 1 -- 172.21.15.12:0/1000542715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a180741c0 msgr2=0x7f3a18199cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:41.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.702+0000 7f3a1f800640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a180741c0 0x7f3a18199cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00e9f0 tx=0x7f3a0c00eeb0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.702+0000 7f3a1f800640 1 -- 172.21.15.12:0/1000542715 shutdown_connections 2024-03-18T00:06:41.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.702+0000 7f3a1f800640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3a000a2880 0x7f3a000a4d40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.702+0000 7f3a1f800640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a181096a0 0x7f3a181a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.702+0000 7f3a1f800640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a180757a0 0x7f3a1819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.702+0000 7f3a1f800640 1 --2- 172.21.15.12:0/1000542715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a180741c0 0x7f3a18199cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.702+0000 7f3a1f800640 1 -- 172.21.15.12:0/1000542715 >> 172.21.15.12:0/1000542715 conn(0x7f3a180fcd80 msgr2=0x7f3a18105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:06:41.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.702+0000 7f3a1f800640 1 -- 172.21.15.12:0/1000542715 shutdown_connections 2024-03-18T00:06:41.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.703+0000 7f3a1f800640 1 -- 172.21.15.12:0/1000542715 wait complete. 2024-03-18T00:06:41.736 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T00:06:41.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:41 smithi079 ceph-mon[36241]: pgmap v677: 129 pgs: 129 active+clean; 238 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 556 op/s 2024-03-18T00:06:41.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:41 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1166507136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:06:41.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:41 smithi191 ceph-mon[36708]: pgmap v677: 129 pgs: 129 active+clean; 238 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 556 op/s 2024-03-18T00:06:41.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:41 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1166507136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:06:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:41 smithi012 ceph-mon[31541]: pgmap v677: 129 pgs: 129 active+clean; 238 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 556 op/s 2024-03-18T00:06:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:41 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1166507136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:06:41.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.989+0000 7fbc06a00640 1 -- 172.21.15.12:0/293669445 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc001096a0 msgr2=0x7fbc0010bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:41.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.989+0000 7fbc06a00640 1 --2- 172.21.15.12:0/293669445 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc001096a0 0x7fbc0010bb30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009ca0 tx=0x7fbbf802c190 comp rx=0 tx=0).stop 2024-03-18T00:06:41.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.989+0000 7fbc06a00640 1 -- 172.21.15.12:0/293669445 shutdown_connections 2024-03-18T00:06:41.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.989+0000 7fbc06a00640 1 --2- 172.21.15.12:0/293669445 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc001096a0 0x7fbc0010bb30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.989+0000 7fbc06a00640 1 --2- 172.21.15.12:0/293669445 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc000741c0 0x7fbc00074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.989+0000 7fbc06a00640 1 --2- 172.21.15.12:0/293669445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc000757a0 0x7fbc00073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.989+0000 7fbc06a00640 1 -- 172.21.15.12:0/293669445 >> 172.21.15.12:0/293669445 conn(0x7fbc000fcd80 msgr2=0x7fbc000ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:06:41.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.989+0000 7fbc06a00640 1 -- 172.21.15.12:0/293669445 shutdown_connections 2024-03-18T00:06:41.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.989+0000 7fbc06a00640 1 -- 172.21.15.12:0/293669445 wait complete. 2024-03-18T00:06:41.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.990+0000 7fbc06a00640 1 Processor -- start 2024-03-18T00:06:41.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.990+0000 7fbc06a00640 1 -- start start 2024-03-18T00:06:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbc06a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc000741c0 0x7fbc00199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbc06a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc000757a0 0x7fbc0019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbc06a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc001096a0 0x7fbc001a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbc06a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0010f680 con 0x7fbc001096a0 2024-03-18T00:06:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbc06a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0010f500 con 0x7fbc000741c0 2024-03-18T00:06:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbc06a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0010f800 con 0x7fbc000757a0 2024-03-18T00:06:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbbffe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc000741c0 0x7fbc00199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbbf7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc000757a0 0x7fbc0019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbbffe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc000741c0 0x7fbc00199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:41794/0 (socket says 172.21.15.12:41794) 2024-03-18T00:06:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbbffe00640 1 -- 172.21.15.12:0/3929970198 learned_addr learned my addr 172.21.15.12:0/3929970198 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:06:41.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbc04a00640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc001096a0 0x7fbc001a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:41.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbbffe00640 1 -- 172.21.15.12:0/3929970198 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc000757a0 msgr2=0x7fbc0019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:41.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbbffe00640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc000757a0 0x7fbc0019a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbbffe00640 1 -- 172.21.15.12:0/3929970198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc001096a0 msgr2=0x7fbc001a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:41.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbbffe00640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc001096a0 0x7fbc001a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:41.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbbffe00640 1 -- 172.21.15.12:0/3929970198 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc001a18f0 con 0x7fbc000741c0 2024-03-18T00:06:41.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.991+0000 7fbc04a00640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc001096a0 0x7fbc001a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:06:41.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.992+0000 7fbbffe00640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc000741c0 0x7fbc00199cb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000ea60 tx=0x7fbbf000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:06:41.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.992+0000 7fbbfd600640 1 -- 172.21.15.12:0/3929970198 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf000cde0 con 0x7fbc000741c0 2024-03-18T00:06:41.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.992+0000 7fbbfd600640 1 -- 172.21.15.12:0/3929970198 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbf00035f0 con 0x7fbc000741c0 2024-03-18T00:06:41.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.992+0000 7fbbfd600640 1 -- 172.21.15.12:0/3929970198 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0010670 con 0x7fbc000741c0 2024-03-18T00:06:41.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.992+0000 7fbc06a00640 1 -- 172.21.15.12:0/3929970198 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc001a1be0 con 0x7fbc000741c0 2024-03-18T00:06:41.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.992+0000 7fbc06a00640 1 -- 172.21.15.12:0/3929970198 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc001a20e0 con 0x7fbc000741c0 2024-03-18T00:06:41.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.993+0000 7fbc06a00640 1 -- 172.21.15.12:0/3929970198 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc00108460 con 0x7fbc000741c0 2024-03-18T00:06:41.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.997+0000 7fbbfd600640 1 -- 172.21.15.12:0/3929970198 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fbbf00037a0 con 0x7fbc000741c0 2024-03-18T00:06:41.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.998+0000 7fbbfd600640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbbe40a2800 0x7fbbe40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:41.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.998+0000 7fbbfd600640 1 -- 172.21.15.12:0/3929970198 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fbbf00cf1d0 con 0x7fbc000741c0 2024-03-18T00:06:42.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.998+0000 7fbbf7e00640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbbe40a2800 0x7fbbe40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:42.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:41.999+0000 7fbbf7e00640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbbe40a2800 0x7fbbe40a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbc0019b1b0 tx=0x7fbbe8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:06:42.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.002+0000 7fbbfd600640 1 -- 172.21.15.12:0/3929970198 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbbf009be90 con 0x7fbc000741c0 2024-03-18T00:06:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.208+0000 7fbc06a00640 1 -- 172.21.15.12:0/3929970198 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbc000624e0 con 0x7fbc000741c0 2024-03-18T00:06:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.209+0000 7fbbfd600640 1 -- 172.21.15.12:0/3929970198 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fbbf009b790 con 0x7fbc000741c0 2024-03-18T00:06:42.210 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-18T00:06:42.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.211+0000 7fbc06a00640 1 -- 172.21.15.12:0/3929970198 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbbe40a2800 msgr2=0x7fbbe40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:42.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.211+0000 7fbc06a00640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbbe40a2800 0x7fbbe40a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbc0019b1b0 tx=0x7fbbe8005f90 comp rx=0 tx=0).stop 2024-03-18T00:06:42.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.211+0000 7fbc06a00640 1 -- 172.21.15.12:0/3929970198 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc000741c0 msgr2=0x7fbc00199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:42.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.211+0000 7fbc06a00640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc000741c0 0x7fbc00199cb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000ea60 tx=0x7fbbf000ef20 comp rx=0 tx=0).stop 2024-03-18T00:06:42.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.211+0000 7fbc06a00640 1 -- 172.21.15.12:0/3929970198 shutdown_connections 2024-03-18T00:06:42.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.211+0000 7fbc06a00640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbbe40a2800 0x7fbbe40a4cc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.211+0000 7fbc06a00640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc001096a0 0x7fbc001a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.211+0000 7fbc06a00640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc000757a0 0x7fbc0019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.211+0000 7fbc06a00640 1 --2- 172.21.15.12:0/3929970198 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc000741c0 0x7fbc00199cb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.211+0000 7fbc06a00640 1 -- 172.21.15.12:0/3929970198 >> 172.21.15.12:0/3929970198 conn(0x7fbc000fcd80 msgr2=0x7fbc00105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:06:42.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.211+0000 7fbc06a00640 1 -- 172.21.15.12:0/3929970198 shutdown_connections 2024-03-18T00:06:42.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.211+0000 7fbc06a00640 1 -- 172.21.15.12:0/3929970198 wait complete. 2024-03-18T00:06:42.230 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T00:06:42.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.436+0000 7fab7d600640 1 -- 172.21.15.12:0/3662734528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab781028d0 msgr2=0x7fab78102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:42.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.436+0000 7fab7d600640 1 --2- 172.21.15.12:0/3662734528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab781028d0 0x7fab78102d50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fab60009900 tx=0x7fab6002bc40 comp rx=0 tx=0).stop 2024-03-18T00:06:42.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.436+0000 7fab7d600640 1 -- 172.21.15.12:0/3662734528 shutdown_connections 2024-03-18T00:06:42.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.436+0000 7fab7d600640 1 --2- 172.21.15.12:0/3662734528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab78103290 0x7fab78109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.436+0000 7fab7d600640 1 --2- 172.21.15.12:0/3662734528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab781028d0 0x7fab78102d50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.436+0000 7fab7d600640 1 --2- 172.21.15.12:0/3662734528 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab781016e0 0x7fab78101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.436+0000 7fab7d600640 1 -- 172.21.15.12:0/3662734528 >> 172.21.15.12:0/3662734528 conn(0x7fab780fcc50 msgr2=0x7fab780ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:06:42.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.436+0000 7fab7d600640 1 -- 172.21.15.12:0/3662734528 shutdown_connections 2024-03-18T00:06:42.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.436+0000 7fab7d600640 1 -- 172.21.15.12:0/3662734528 wait complete. 2024-03-18T00:06:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.437+0000 7fab7d600640 1 Processor -- start 2024-03-18T00:06:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.438+0000 7fab7d600640 1 -- start start 2024-03-18T00:06:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.438+0000 7fab7d600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab781016e0 0x7fab78075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.438+0000 7fab7d600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab781028d0 0x7fab78073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.438+0000 7fab7d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab78103290 0x7fab78074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.438+0000 7fab7d600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fab7810d390 con 0x7fab781016e0 2024-03-18T00:06:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.438+0000 7fab7d600640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fab7810d210 con 0x7fab781028d0 2024-03-18T00:06:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.438+0000 7fab7d600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fab7810d510 con 0x7fab78103290 2024-03-18T00:06:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.438+0000 7fab77e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab781016e0 0x7fab78075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.438+0000 7fab77e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab781016e0 0x7fab78075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56450/0 (socket says 172.21.15.12:56450) 2024-03-18T00:06:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.438+0000 7fab6f400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab781028d0 0x7fab78073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.438+0000 7fab77e00640 1 -- 172.21.15.12:0/3983744729 learned_addr learned my addr 172.21.15.12:0/3983744729 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:06:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.438+0000 7fab7cc00640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab78103290 0x7fab78074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.439+0000 7fab6f400640 1 -- 172.21.15.12:0/3983744729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab78103290 msgr2=0x7fab78074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.439+0000 7fab6f400640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab78103290 0x7fab78074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.439+0000 7fab6f400640 1 -- 172.21.15.12:0/3983744729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab781016e0 msgr2=0x7fab78075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.439+0000 7fab6f400640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab781016e0 0x7fab78075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.439+0000 7fab6f400640 1 -- 172.21.15.12:0/3983744729 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab780749a0 con 0x7fab781028d0 2024-03-18T00:06:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.439+0000 7fab77e00640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab781016e0 0x7fab78075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:06:42.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.439+0000 7fab6f400640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab781028d0 0x7fab78073bd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fab60036040 tx=0x7fab6002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:06:42.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.439+0000 7fab75600640 1 -- 172.21.15.12:0/3983744729 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60041070 con 0x7fab781028d0 2024-03-18T00:06:42.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.439+0000 7fab75600640 1 -- 172.21.15.12:0/3983744729 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab60004450 con 0x7fab781028d0 2024-03-18T00:06:42.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.439+0000 7fab75600640 1 -- 172.21.15.12:0/3983744729 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60003e60 con 0x7fab781028d0 2024-03-18T00:06:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.439+0000 7fab7d600640 1 -- 172.21.15.12:0/3983744729 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab781a1a60 con 0x7fab781028d0 2024-03-18T00:06:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.440+0000 7fab7d600640 1 -- 172.21.15.12:0/3983744729 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab781a1f30 con 0x7fab781028d0 2024-03-18T00:06:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.441+0000 7fab7d600640 1 -- 172.21.15.12:0/3983744729 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab78108a00 con 0x7fab781028d0 2024-03-18T00:06:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.442+0000 7fab75600640 1 -- 172.21.15.12:0/3983744729 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fab60044070 con 0x7fab781028d0 2024-03-18T00:06:42.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.444+0000 7fab75600640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fab5c0a2800 0x7fab5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:42.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.444+0000 7fab75600640 1 -- 172.21.15.12:0/3983744729 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fab600f08f0 con 0x7fab781028d0 2024-03-18T00:06:42.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.444+0000 7fab77e00640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fab5c0a2800 0x7fab5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:42.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.445+0000 7fab77e00640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fab5c0a2800 0x7fab5c0a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fab680058c0 tx=0x7fab68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:06:42.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.448+0000 7fab75600640 1 -- 172.21.15.12:0/3983744729 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fab600bcf60 con 0x7fab781028d0 2024-03-18T00:06:42.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.673+0000 7fab7d600640 1 -- 172.21.15.12:0/3983744729 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fab780623e0 con 0x7fab781028d0 2024-03-18T00:06:42.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.674+0000 7fab75600640 1 -- 172.21.15.12:0/3983744729 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fab600c1000 con 0x7fab781028d0 2024-03-18T00:06:42.675 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-18T00:06:42.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.676+0000 7fab7d600640 1 -- 172.21.15.12:0/3983744729 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fab5c0a2800 msgr2=0x7fab5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:42.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.676+0000 7fab7d600640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fab5c0a2800 0x7fab5c0a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fab680058c0 tx=0x7fab68005f90 comp rx=0 tx=0).stop 2024-03-18T00:06:42.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.676+0000 7fab7d600640 1 -- 172.21.15.12:0/3983744729 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab781028d0 msgr2=0x7fab78073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:42.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.676+0000 7fab7d600640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab781028d0 0x7fab78073bd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fab60036040 tx=0x7fab6002c660 comp rx=0 tx=0).stop 2024-03-18T00:06:42.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.676+0000 7fab7d600640 1 -- 172.21.15.12:0/3983744729 shutdown_connections 2024-03-18T00:06:42.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.676+0000 7fab7d600640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fab5c0a2800 0x7fab5c0a4cc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.676+0000 7fab7d600640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab78103290 0x7fab78074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.676+0000 7fab7d600640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab781028d0 0x7fab78073bd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.676+0000 7fab7d600640 1 --2- 172.21.15.12:0/3983744729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab781016e0 0x7fab78075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.676+0000 7fab7d600640 1 -- 172.21.15.12:0/3983744729 >> 172.21.15.12:0/3983744729 conn(0x7fab780fcc50 msgr2=0x7fab78105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:06:42.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.676+0000 7fab7d600640 1 -- 172.21.15.12:0/3983744729 shutdown_connections 2024-03-18T00:06:42.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.676+0000 7fab7d600640 1 -- 172.21.15.12:0/3983744729 wait complete. 2024-03-18T00:06:42.695 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T00:06:42.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:42 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:06:42.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:42 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1000542715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:06:42.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:42 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3929970198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:06:42.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.882+0000 7f5cca800640 1 -- 172.21.15.12:0/3462877513 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5cc41017f0 msgr2=0x7f5cc4101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:42.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.882+0000 7f5cca800640 1 --2- 172.21.15.12:0/3462877513 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5cc41017f0 0x7f5cc4101c10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009900 tx=0x7f5cb402bc40 comp rx=0 tx=0).stop 2024-03-18T00:06:42.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.882+0000 7f5cca800640 1 -- 172.21.15.12:0/3462877513 shutdown_connections 2024-03-18T00:06:42.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.882+0000 7f5cca800640 1 --2- 172.21.15.12:0/3462877513 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc41033a0 0x7f5cc4109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.882+0000 7f5cca800640 1 --2- 172.21.15.12:0/3462877513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cc41029e0 0x7f5cc4102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.882+0000 7f5cca800640 1 --2- 172.21.15.12:0/3462877513 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5cc41017f0 0x7f5cc4101c10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.882+0000 7f5cca800640 1 -- 172.21.15.12:0/3462877513 >> 172.21.15.12:0/3462877513 conn(0x7f5cc40fcd80 msgr2=0x7f5cc40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:06:42.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.882+0000 7f5cca800640 1 -- 172.21.15.12:0/3462877513 shutdown_connections 2024-03-18T00:06:42.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.882+0000 7f5cca800640 1 -- 172.21.15.12:0/3462877513 wait complete. 2024-03-18T00:06:42.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.883+0000 7f5cca800640 1 Processor -- start 2024-03-18T00:06:42.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.883+0000 7f5cca800640 1 -- start start 2024-03-18T00:06:42.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.884+0000 7f5cca800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cc41017f0 0x7f5cc4075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:42.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.884+0000 7f5cca800640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5cc41029e0 0x7f5cc4073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:42.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.884+0000 7f5cca800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc41033a0 0x7f5cc4074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:42.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.884+0000 7f5cca800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc410d4b0 con 0x7f5cc41033a0 2024-03-18T00:06:42.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.884+0000 7f5cca800640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc410d330 con 0x7f5cc41029e0 2024-03-18T00:06:42.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.884+0000 7f5cca800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc410d630 con 0x7f5cc41017f0 2024-03-18T00:06:42.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.884+0000 7f5cc3400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cc41017f0 0x7f5cc4075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:42.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.884+0000 7f5cc3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc41033a0 0x7f5cc4074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:42.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.884+0000 7f5cc2a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5cc41029e0 0x7f5cc4073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:42.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.884+0000 7f5cc3400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cc41017f0 0x7f5cc4075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:35134/0 (socket says 172.21.15.12:35134) 2024-03-18T00:06:42.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.884+0000 7f5cc3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc41033a0 0x7f5cc4074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56466/0 (socket says 172.21.15.12:56466) 2024-03-18T00:06:42.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.884+0000 7f5cc3400640 1 -- 172.21.15.12:0/1953950439 learned_addr learned my addr 172.21.15.12:0/1953950439 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:06:42.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.885+0000 7f5cc3400640 1 -- 172.21.15.12:0/1953950439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5cc41029e0 msgr2=0x7f5cc4073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:42.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.885+0000 7f5cc3400640 1 --2- 172.21.15.12:0/1953950439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5cc41029e0 0x7f5cc4073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.885+0000 7f5cc3400640 1 -- 172.21.15.12:0/1953950439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc41033a0 msgr2=0x7f5cc4074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:42.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.885+0000 7f5cc3400640 1 --2- 172.21.15.12:0/1953950439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc41033a0 0x7f5cc4074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:42.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.885+0000 7f5cc3400640 1 -- 172.21.15.12:0/1953950439 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc40749a0 con 0x7f5cc41017f0 2024-03-18T00:06:42.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.885+0000 7f5cc2a00640 1 --2- 172.21.15.12:0/1953950439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5cc41029e0 0x7f5cc4073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:06:42.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.885+0000 7f5cc3400640 1 --2- 172.21.15.12:0/1953950439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cc41017f0 0x7f5cc4075590 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5cb402efd0 tx=0x7f5cb402c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:06:42.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.885+0000 7f5cbbe00640 1 -- 172.21.15.12:0/1953950439 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4041070 con 0x7f5cc41017f0 2024-03-18T00:06:42.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.885+0000 7f5cbbe00640 1 -- 172.21.15.12:0/1953950439 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cb4004450 con 0x7f5cc41017f0 2024-03-18T00:06:42.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.885+0000 7f5cca800640 1 -- 172.21.15.12:0/1953950439 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc41a1c70 con 0x7f5cc41017f0 2024-03-18T00:06:42.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.885+0000 7f5cbbe00640 1 -- 172.21.15.12:0/1953950439 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4003d90 con 0x7f5cc41017f0 2024-03-18T00:06:42.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.885+0000 7f5cca800640 1 -- 172.21.15.12:0/1953950439 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc41a2190 con 0x7f5cc41017f0 2024-03-18T00:06:42.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.886+0000 7f5cca800640 1 -- 172.21.15.12:0/1953950439 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cc4108b30 con 0x7f5cc41017f0 2024-03-18T00:06:42.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.888+0000 7f5cbbe00640 1 -- 172.21.15.12:0/1953950439 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f5cb4044070 con 0x7f5cc41017f0 2024-03-18T00:06:42.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.891+0000 7f5cbbe00640 1 --2- 172.21.15.12:0/1953950439 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5ca80a2880 0x7f5ca80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:42.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.891+0000 7f5cbbe00640 1 -- 172.21.15.12:0/1953950439 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f5cb40f0c20 con 0x7f5cc41017f0 2024-03-18T00:06:42.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.891+0000 7f5cc2a00640 1 --2- 172.21.15.12:0/1953950439 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5ca80a2880 0x7f5ca80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:42.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.892+0000 7f5cc2a00640 1 --2- 172.21.15.12:0/1953950439 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5ca80a2880 0x7f5ca80a4d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4075300 tx=0x7f5cb0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:06:42.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:42.895+0000 7f5cbbe00640 1 -- 172.21.15.12:0/1953950439 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5cb40bd8e0 con 0x7f5cc41017f0 2024-03-18T00:06:42.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:42 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:06:42.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:42 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1000542715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:06:42.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:42 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3929970198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:06:43.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:42 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:06:43.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:42 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1000542715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:06:43.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:42 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3929970198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:06:43.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.168+0000 7f5cca800640 1 -- 172.21.15.12:0/1953950439 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5cc40623e0 con 0x7f5cc41017f0 2024-03-18T00:06:43.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.169+0000 7f5cbbe00640 1 -- 172.21.15.12:0/1953950439 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f5cb40bd1e0 con 0x7f5cc41017f0 2024-03-18T00:06:43.170 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-18T00:06:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.172+0000 7f5cca800640 1 -- 172.21.15.12:0/1953950439 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5ca80a2880 msgr2=0x7f5ca80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.172+0000 7f5cca800640 1 --2- 172.21.15.12:0/1953950439 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5ca80a2880 0x7f5ca80a4d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4075300 tx=0x7f5cb0005f90 comp rx=0 tx=0).stop 2024-03-18T00:06:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.172+0000 7f5cca800640 1 -- 172.21.15.12:0/1953950439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cc41017f0 msgr2=0x7f5cc4075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.172+0000 7f5cca800640 1 --2- 172.21.15.12:0/1953950439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cc41017f0 0x7f5cc4075590 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5cb402efd0 tx=0x7f5cb402c660 comp rx=0 tx=0).stop 2024-03-18T00:06:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.172+0000 7f5cca800640 1 -- 172.21.15.12:0/1953950439 shutdown_connections 2024-03-18T00:06:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.172+0000 7f5cca800640 1 --2- 172.21.15.12:0/1953950439 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f5ca80a2880 0x7f5ca80a4d40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.172+0000 7f5cca800640 1 --2- 172.21.15.12:0/1953950439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc41033a0 0x7f5cc4074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.172+0000 7f5cca800640 1 --2- 172.21.15.12:0/1953950439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5cc41029e0 0x7f5cc4073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.172+0000 7f5cca800640 1 --2- 172.21.15.12:0/1953950439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cc41017f0 0x7f5cc4075590 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.172+0000 7f5cca800640 1 -- 172.21.15.12:0/1953950439 >> 172.21.15.12:0/1953950439 conn(0x7f5cc40fcd80 msgr2=0x7f5cc4105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:06:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.172+0000 7f5cca800640 1 -- 172.21.15.12:0/1953950439 shutdown_connections 2024-03-18T00:06:43.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.172+0000 7f5cca800640 1 -- 172.21.15.12:0/1953950439 wait complete. 2024-03-18T00:06:43.201 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-18T00:06:43.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.444+0000 7fea6b600640 1 -- 172.21.15.12:0/2876750110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea641022b0 msgr2=0x7fea64108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:43.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.444+0000 7fea6b600640 1 --2- 172.21.15.12:0/2876750110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea641022b0 0x7fea64108b40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fea5c009ca0 tx=0x7fea5c02c100 comp rx=0 tx=0).stop 2024-03-18T00:06:43.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.444+0000 7fea6b600640 1 -- 172.21.15.12:0/2876750110 shutdown_connections 2024-03-18T00:06:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.444+0000 7fea6b600640 1 --2- 172.21.15.12:0/2876750110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea641022b0 0x7fea64108b40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.444+0000 7fea6b600640 1 --2- 172.21.15.12:0/2876750110 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea641018f0 0x7fea64101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.444+0000 7fea6b600640 1 --2- 172.21.15.12:0/2876750110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea64100700 0x7fea64100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.444+0000 7fea6b600640 1 -- 172.21.15.12:0/2876750110 >> 172.21.15.12:0/2876750110 conn(0x7fea640fbc70 msgr2=0x7fea640fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:06:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.444+0000 7fea6b600640 1 -- 172.21.15.12:0/2876750110 shutdown_connections 2024-03-18T00:06:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.444+0000 7fea6b600640 1 -- 172.21.15.12:0/2876750110 wait complete. 2024-03-18T00:06:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.445+0000 7fea6b600640 1 Processor -- start 2024-03-18T00:06:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.445+0000 7fea6b600640 1 -- start start 2024-03-18T00:06:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea6b600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea64100700 0x7fea641953d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea6b600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea641018f0 0x7fea64195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea6b600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea641022b0 0x7fea64191f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea6b600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fea6410c380 con 0x7fea641018f0 2024-03-18T00:06:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea6b600640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fea6410c200 con 0x7fea641022b0 2024-03-18T00:06:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea6b600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fea6410c500 con 0x7fea64100700 2024-03-18T00:06:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea68c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea64100700 0x7fea641953d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea68c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea64100700 0x7fea641953d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:35152/0 (socket says 172.21.15.12:35152) 2024-03-18T00:06:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea68c00640 1 -- 172.21.15.12:0/1018618744 learned_addr learned my addr 172.21.15.12:0/1018618744 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:06:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea5be00640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea641018f0 0x7fea64195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea69600640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea641022b0 0x7fea64191f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea68c00640 1 -- 172.21.15.12:0/1018618744 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea641022b0 msgr2=0x7fea64191f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea68c00640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea641022b0 0x7fea64191f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea68c00640 1 -- 172.21.15.12:0/1018618744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea641018f0 msgr2=0x7fea64195910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea68c00640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea641018f0 0x7fea64195910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea68c00640 1 -- 172.21.15.12:0/1018618744 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea64192800 con 0x7fea64100700 2024-03-18T00:06:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.446+0000 7fea5be00640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea641018f0 0x7fea64195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:06:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.447+0000 7fea68c00640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea64100700 0x7fea641953d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fea5400e9c0 tx=0x7fea5400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:06:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.447+0000 7fea62000640 1 -- 172.21.15.12:0/1018618744 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5400cd80 con 0x7fea64100700 2024-03-18T00:06:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.447+0000 7fea6b600640 1 -- 172.21.15.12:0/1018618744 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea641a6140 con 0x7fea64100700 2024-03-18T00:06:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.447+0000 7fea62000640 1 -- 172.21.15.12:0/1018618744 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea540035f0 con 0x7fea64100700 2024-03-18T00:06:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.447+0000 7fea62000640 1 -- 172.21.15.12:0/1018618744 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea54010690 con 0x7fea64100700 2024-03-18T00:06:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.447+0000 7fea6b600640 1 -- 172.21.15.12:0/1018618744 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea641a65a0 con 0x7fea64100700 2024-03-18T00:06:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.448+0000 7fea6b600640 1 -- 172.21.15.12:0/1018618744 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea641a6860 con 0x7fea64100700 2024-03-18T00:06:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.450+0000 7fea62000640 1 -- 172.21.15.12:0/1018618744 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fea540037a0 con 0x7fea64100700 2024-03-18T00:06:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.452+0000 7fea62000640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fea480a2800 0x7fea480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:06:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.452+0000 7fea62000640 1 -- 172.21.15.12:0/1018618744 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fea540cf2a0 con 0x7fea64100700 2024-03-18T00:06:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.452+0000 7fea5be00640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fea480a2800 0x7fea480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:06:43.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.452+0000 7fea5be00640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fea480a2800 0x7fea480a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fea64193350 tx=0x7fea4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:06:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.457+0000 7fea62000640 1 -- 172.21.15.12:0/1018618744 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fea5409bf60 con 0x7fea64100700 2024-03-18T00:06:43.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.751+0000 7fea6b600640 1 -- 172.21.15.12:0/1018618744 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fea641a6ad0 con 0x7fea64100700 2024-03-18T00:06:43.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.761+0000 7fea62000640 1 -- 172.21.15.12:0/1018618744 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+739831 (secure 0 0 0) 0x7fea5409b860 con 0x7fea64100700 2024-03-18T00:06:43.773 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-18T00:06:43.774 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-18T00:06:43.774 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 20, 2024-03-18T00:06:43.774 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.774 INFO:teuthology.orchestra.run.smithi012.stdout: "active_gid": 14152, 2024-03-18T00:06:43.774 INFO:teuthology.orchestra.run.smithi012.stdout: "active_name": "x", 2024-03-18T00:06:43.774 INFO:teuthology.orchestra.run.smithi012.stdout: "active_addrs": { 2024-03-18T00:06:43.774 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-03-18T00:06:43.774 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.774 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-03-18T00:06:43.774 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6800", 2024-03-18T00:06:43.774 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3179836366 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6801", 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3179836366 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: "active_addr": "172.21.15.12:6801/3179836366", 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: "active_change": "2024-03-17T23:46:53.956106+0000", 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: "active_mgr_features": 4540701547738038271, 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: "available": true, 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: "standbys": [ 2024-03-18T00:06:43.775 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.776 INFO:teuthology.orchestra.run.smithi012.stdout: "gid": 24109, 2024-03-18T00:06:43.776 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "z", 2024-03-18T00:06:43.776 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr_features": 4540701547738038271, 2024-03-18T00:06:43.776 INFO:teuthology.orchestra.run.smithi012.stdout: "available_modules": [ 2024-03-18T00:06:43.776 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.776 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "alerts", 2024-03-18T00:06:43.776 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:43.776 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:43.776 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:43.776 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:43.776 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:43.776 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.776 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.777 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.777 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:43.777 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.777 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.777 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.777 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T00:06:43.777 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.777 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.777 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.777 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.777 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:43.777 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:43.777 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.778 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.778 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.778 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.778 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.778 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.778 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.778 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.778 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.778 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.778 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.778 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.778 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.778 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.779 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.780 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.780 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.780 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.780 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.780 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.780 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:43.780 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:43.780 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.780 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.780 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.780 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:43.780 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.780 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.781 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.782 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_destination": { 2024-03-18T00:06:43.783 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_destination", 2024-03-18T00:06:43.783 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.783 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.783 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.783 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.783 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.783 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.783 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.783 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email address to send alerts to", 2024-03-18T00:06:43.783 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.783 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.783 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.783 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.784 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_from_name": { 2024-03-18T00:06:43.784 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_from_name", 2024-03-18T00:06:43.784 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.784 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.784 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.784 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "Ceph", 2024-03-18T00:06:43.784 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.784 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.784 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.784 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email From: name", 2024-03-18T00:06:43.785 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.785 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.785 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.785 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_host": { 2024-03-18T00:06:43.785 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_host", 2024-03-18T00:06:43.785 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.785 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.785 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.785 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.785 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.785 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.785 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP server", 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_password": { 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_password", 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.786 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Password to authenticate with", 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_port": { 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_port", 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "465", 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.787 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP port", 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_sender": { 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_sender", 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.788 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP envelope sender", 2024-03-18T00:06:43.789 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.789 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.789 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.789 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_ssl": { 2024-03-18T00:06:43.789 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_ssl", 2024-03-18T00:06:43.789 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.789 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.789 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.789 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.789 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.789 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.790 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.790 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T00:06:43.790 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.790 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.790 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.790 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_user": { 2024-03-18T00:06:43.790 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_user", 2024-03-18T00:06:43.790 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.790 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.790 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.790 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.790 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "User to authenticate as", 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "balancer", 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:43.791 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: "active": { 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "active", 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_time": { 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_time", 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0000", 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.794 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_weekday": { 2024-03-18T00:06:43.794 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_weekday", 2024-03-18T00:06:43.794 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:43.794 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.794 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.794 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:43.794 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-03-18T00:06:43.794 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-03-18T00:06:43.794 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.794 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T00:06:43.794 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T00:06:43.794 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.794 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_max_iterations": { 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_max_iterations", 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "25", 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "250", 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.795 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_metrics": { 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_metrics", 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_step": { 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_step", 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.5", 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0.001", 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "0.999", 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "aggressiveness of optimization", 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.798 INFO:teuthology.orchestra.run.smithi012.stdout: "end_time": { 2024-03-18T00:06:43.798 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_time", 2024-03-18T00:06:43.798 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.798 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.798 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.798 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2359", 2024-03-18T00:06:43.798 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.798 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.798 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.798 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T00:06:43.798 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T00:06:43.798 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.798 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: "end_weekday": { 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_weekday", 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.799 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.800 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.801 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.801 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.801 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.801 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.801 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.801 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.801 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.801 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:43.801 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:43.801 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.801 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.801 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.802 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.802 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.802 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.802 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.802 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.802 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.802 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.802 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.802 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:43.802 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:43.802 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.803 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.804 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "min_score": { 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_score", 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T00:06:43.805 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.806 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.806 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.806 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.806 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-03-18T00:06:43.806 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-03-18T00:06:43.806 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.806 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.806 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.806 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "upmap", 2024-03-18T00:06:43.806 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.806 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.806 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.806 INFO:teuthology.orchestra.run.smithi012.stdout: "crush-compat", 2024-03-18T00:06:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: "none", 2024-03-18T00:06:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: "read", 2024-03-18T00:06:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap", 2024-03-18T00:06:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap-read" 2024-03-18T00:06:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Balancer mode", 2024-03-18T00:06:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_ids": { 2024-03-18T00:06:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_ids", 2024-03-18T00:06:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:43.808 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_deviation": { 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_deviation", 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.809 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.810 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.810 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-18T00:06:43.810 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-18T00:06:43.810 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.810 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.810 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T00:06:43.810 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T00:06:43.810 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.810 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.810 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_optimizations": { 2024-03-18T00:06:43.810 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_optimizations", 2024-03-18T00:06:43.810 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.811 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm", 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_down_multiplier": { 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_down_multiplier", 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T00:06:43.812 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_refresh_rate": { 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_refresh_rate", 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "20", 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T00:06:43.813 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.814 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.814 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.814 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.814 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_starting_port": { 2024-03-18T00:06:43.814 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_starting_port", 2024-03-18T00:06:43.814 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.814 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.814 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.814 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4721", 2024-03-18T00:06:43.814 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.814 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.814 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.814 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T00:06:43.815 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.815 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.815 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.815 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_ptrace": { 2024-03-18T00:06:43.815 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_ptrace", 2024-03-18T00:06:43.815 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.815 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.815 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.815 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.815 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.815 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_interval": { 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_interval", 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.816 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.817 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.817 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T00:06:43.817 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.817 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.817 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.817 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.817 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_memory_target_ratio": { 2024-03-18T00:06:43.817 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T00:06:43.817 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:43.817 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.817 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.817 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.7", 2024-03-18T00:06:43.817 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.818 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.818 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.818 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T00:06:43.818 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.818 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.818 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.818 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.818 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm_log_destination": { 2024-03-18T00:06:43.818 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm_log_destination", 2024-03-18T00:06:43.818 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.818 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.818 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.818 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: "file", 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: "file,syslog", 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: "syslog" 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: "cgroups_split": { 2024-03-18T00:06:43.819 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cgroups_split", 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: "config_checks_enabled": { 2024-03-18T00:06:43.820 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_checks_enabled", 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "config_dashboard": { 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_dashboard", 2024-03-18T00:06:43.821 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T00:06:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_alertmanager": { 2024-03-18T00:06:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_alertmanager", 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_base": { 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_base", 2024-03-18T00:06:43.823 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Container image name, without the tag", 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_elasticsearch": { 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_elasticsearch", 2024-03-18T00:06:43.824 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "elasticsearch container image", 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_grafana": { 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_grafana", 2024-03-18T00:06:43.825 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.826 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.826 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.826 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T00:06:43.826 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.826 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.826 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.826 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-18T00:06:43.826 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.826 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.826 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.826 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_haproxy": { 2024-03-18T00:06:43.826 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_haproxy", 2024-03-18T00:06:43.827 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.827 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.827 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.827 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T00:06:43.827 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.827 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.827 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.827 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "HAproxy container image", 2024-03-18T00:06:43.827 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.827 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.827 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_agent": { 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_agent", 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger agent container image", 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_collector": { 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_collector", 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger collector container image", 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_query": { 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_query", 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger query container image", 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_keepalived": { 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_keepalived", 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Keepalived container image", 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.832 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_loki": { 2024-03-18T00:06:43.832 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_loki", 2024-03-18T00:06:43.832 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.832 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.832 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.832 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T00:06:43.832 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.832 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.832 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.832 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Loki container image", 2024-03-18T00:06:43.832 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.832 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.832 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.833 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.833 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_node_exporter": { 2024-03-18T00:06:43.833 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_node_exporter", 2024-03-18T00:06:43.833 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.833 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.833 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.833 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T00:06:43.833 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.833 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.833 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.833 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-18T00:06:43.833 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.833 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.834 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.834 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.834 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_nvmeof": { 2024-03-18T00:06:43.834 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_nvmeof", 2024-03-18T00:06:43.834 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.834 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.834 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.834 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T00:06:43.834 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.834 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.834 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.834 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Nvme-of container image", 2024-03-18T00:06:43.834 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_prometheus": { 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_prometheus", 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-18T00:06:43.835 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_promtail": { 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_promtail", 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Promtail container image", 2024-03-18T00:06:43.836 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_snmp_gateway": { 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_snmp_gateway", 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SNMP Gateway container image", 2024-03-18T00:06:43.837 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: "container_init": { 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_init", 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T00:06:43.838 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.839 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.839 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.839 INFO:teuthology.orchestra.run.smithi012.stdout: "daemon_cache_timeout": { 2024-03-18T00:06:43.839 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "daemon_cache_timeout", 2024-03-18T00:06:43.839 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.839 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.839 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.839 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:43.839 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.839 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.839 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.839 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: "default_cephadm_command_timeout": { 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "900", 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.840 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: "default_registry": { 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_registry", 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io", 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.841 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: "device_cache_timeout": { 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_cache_timeout", 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1800", 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.842 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache device inventory", 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: "device_enhanced_scan": { 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_enhanced_scan", 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.843 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: "facts_cache_timeout": { 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "facts_cache_timeout", 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.844 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache host facts data", 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "host_check_interval": { 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "host_check_interval", 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to perform a host check", 2024-03-18T00:06:43.845 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.846 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.846 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.846 INFO:teuthology.orchestra.run.smithi012.stdout: "hw_monitoring": { 2024-03-18T00:06:43.846 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hw_monitoring", 2024-03-18T00:06:43.846 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.846 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.846 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.846 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.846 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.850 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.850 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.850 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T00:06:43.850 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.850 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.850 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.850 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.850 INFO:teuthology.orchestra.run.smithi012.stdout: "inventory_list_all": { 2024-03-18T00:06:43.850 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "inventory_list_all", 2024-03-18T00:06:43.850 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.850 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.850 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.851 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.852 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: "log_refresh_metadata": { 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_refresh_metadata", 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.853 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.854 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.855 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.855 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:43.855 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:43.855 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.855 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.855 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.855 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:43.855 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.855 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.855 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.855 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.855 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.856 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "*", 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: "max_count_per_host": { 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_count_per_host", 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of daemons per service per host", 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "max_osd_draining_count": { 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_osd_draining_count", 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.860 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: "migration_current": { 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "migration_current", 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.861 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "internal - do not modify", 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.862 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "root", 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm-package", 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: "root" 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: "oob_default_addr": { 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "oob_default_addr", 2024-03-18T00:06:43.863 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "169.254.1.1", 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: "prometheus_alerts_path": { 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus_alerts_path", 2024-03-18T00:06:43.864 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_insecure": { 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_insecure", 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.865 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_password": { 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_password", 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.866 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.867 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.867 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.867 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.867 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.867 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.867 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T00:06:43.867 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.867 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.867 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.867 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_url": { 2024-03-18T00:06:43.867 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_url", 2024-03-18T00:06:43.867 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_username": { 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_username", 2024-03-18T00:06:43.868 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: "secure_monitoring_stack": { 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "secure_monitoring_stack", 2024-03-18T00:06:43.869 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: "service_discovery_port": { 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "service_discovery_port", 2024-03-18T00:06:43.870 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8765", 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "cephadm service discovery port", 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "ssh_config_file": { 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssh_config_file", 2024-03-18T00:06:43.871 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "use_agent": { 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_agent", 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.872 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: "use_repo_digest": { 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_repo_digest", 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.873 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_failed_host_check": { 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_failed_host_check", 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_daemons": { 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_daemons", 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.875 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_hosts": { 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_hosts", 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.876 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crash", 2024-03-18T00:06:43.877 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.878 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.879 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.880 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.880 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.880 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.880 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.880 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.880 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.880 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.880 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.880 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:43.880 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:43.880 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.880 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.880 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.881 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.882 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.883 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.883 INFO:teuthology.orchestra.run.smithi012.stdout: "retain_interval": { 2024-03-18T00:06:43.883 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retain_interval", 2024-03-18T00:06:43.883 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.883 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.883 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.883 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "31536000", 2024-03-18T00:06:43.883 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.883 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.883 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.883 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T00:06:43.883 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_recent_interval": { 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_recent_interval", 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1209600", 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T00:06:43.884 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.885 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.885 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.885 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.885 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.885 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.885 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.885 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dashboard", 2024-03-18T00:06:43.885 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:43.885 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:43.885 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:43.885 INFO:teuthology.orchestra.run.smithi012.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T00:06:43.885 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T00:06:43.885 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T00:06:43.886 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.887 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_ENABLED": { 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.888 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.889 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.890 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.891 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.892 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.893 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.893 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.893 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.893 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.893 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.893 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.893 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.893 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.893 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T00:06:43.893 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T00:06:43.893 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.893 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.894 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.894 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.894 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.894 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.894 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.894 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.894 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.894 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.894 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.894 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.894 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T00:06:43.894 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T00:06:43.894 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T00:06:43.895 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T00:06:43.896 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.897 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.898 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.898 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.898 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.898 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.898 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.898 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.898 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.898 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.898 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.898 INFO:teuthology.orchestra.run.smithi012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T00:06:43.898 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T00:06:43.898 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T00:06:43.899 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.900 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_URL": { 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_URL", 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.901 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.902 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.903 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.904 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.905 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.906 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.906 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.906 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.906 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.906 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.906 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.906 INFO:teuthology.orchestra.run.smithi012.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T00:06:43.906 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T00:06:43.906 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.906 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.906 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.907 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "{}", 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.908 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.909 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.909 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.909 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.909 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.909 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.909 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.909 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.909 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.909 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T00:06:43.909 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T00:06:43.909 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.909 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.910 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.911 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.912 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.913 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.914 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.915 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.916 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.917 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.918 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T00:06:43.919 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.919 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.919 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.919 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.919 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.919 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.919 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.919 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.919 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T00:06:43.919 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T00:06:43.919 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.919 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.920 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8", 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.921 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "45", 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.922 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.923 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.924 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.925 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.926 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.927 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.928 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.929 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: "cross_origin_url": { 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cross_origin_url", 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.930 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: "crt_file": { 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crt_file", 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.931 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "debug": { 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "debug", 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.932 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.933 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.933 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.933 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.933 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable/disable debug options", 2024-03-18T00:06:43.933 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.933 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.933 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.933 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.933 INFO:teuthology.orchestra.run.smithi012.stdout: "jwt_token_ttl": { 2024-03-18T00:06:43.933 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "jwt_token_ttl", 2024-03-18T00:06:43.933 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.933 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "28800", 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.934 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.935 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.936 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.937 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.937 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.937 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:43.937 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:43.937 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.937 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.937 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.937 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.937 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.937 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.937 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.937 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.938 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:43.939 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: "motd": { 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "motd", 2024-03-18T00:06:43.940 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "The message of the day", 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_resolve_ip_addr": { 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T00:06:43.941 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-18T00:06:43.942 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.943 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8080", 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.944 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl_server_port": { 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl_server_port", 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.945 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.946 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8443", 2024-03-18T00:06:43.946 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.946 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.946 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.946 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.946 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.946 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.946 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.946 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-03-18T00:06:43.946 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-03-18T00:06:43.946 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "redirect", 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect" 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.948 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: "url_prefix": { 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url_prefix", 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-03-18T00:06:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_monitoring": { 2024-03-18T00:06:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_monitoring", 2024-03-18T00:06:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "monitor device health metrics", 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.951 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.952 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.952 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.952 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.952 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.952 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.952 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.952 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.952 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.952 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.952 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.952 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.952 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.953 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.954 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.955 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: "mark_out_threshold": { 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mark_out_threshold", 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2419200", 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.956 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_name": { 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_name", 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "device_health_metrics", 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.957 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.958 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T00:06:43.958 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.958 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.958 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.958 INFO:teuthology.orchestra.run.smithi012.stdout: "retention_period": { 2024-03-18T00:06:43.958 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retention_period", 2024-03-18T00:06:43.958 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.958 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.958 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.958 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15552000", 2024-03-18T00:06:43.958 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.958 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain device health metrics", 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_frequency": { 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_frequency", 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.959 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.960 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.960 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T00:06:43.960 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.960 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.960 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.960 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.960 INFO:teuthology.orchestra.run.smithi012.stdout: "self_heal": { 2024-03-18T00:06:43.960 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "self_heal", 2024-03-18T00:06:43.960 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.960 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.960 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.960 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.961 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_threshold": { 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_threshold", 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.962 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7257600", 2024-03-18T00:06:43.963 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.963 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.963 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.963 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T00:06:43.963 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.963 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.963 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.963 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.963 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.963 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.963 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.963 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "diskprediction_local", 2024-03-18T00:06:43.963 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.964 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.965 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.966 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.966 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.966 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.966 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.966 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.966 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.966 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.966 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.966 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.966 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:43.966 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:43.966 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.966 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.967 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.967 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:43.967 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.967 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.967 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.967 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.967 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.967 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.967 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.967 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.967 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.967 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.967 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.968 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: "predict_interval": { 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predict_interval", 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.969 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: "predictor_model": { 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predictor_model", 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "prophetstor", 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.970 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.971 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "influx", 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": false, 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "influxdb python module not found", 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: "batch_size": { 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "batch_size", 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.972 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5000", 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: "database": { 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "database", 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.973 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.974 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.974 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "ceph", 2024-03-18T00:06:43.974 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.974 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.974 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.974 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T00:06:43.974 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.974 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.974 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.974 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.974 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": { 2024-03-18T00:06:43.974 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hostname", 2024-03-18T00:06:43.974 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server hostname", 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:43.975 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "30", 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "5", 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.976 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.977 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.977 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.977 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.977 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.977 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.977 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.977 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.977 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.977 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.977 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.977 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.977 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.977 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.978 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.978 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.978 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.978 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.978 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:43.978 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:43.978 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.978 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.979 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.979 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.979 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.979 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.979 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.979 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.979 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.979 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.979 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.979 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.979 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:43.979 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.980 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.981 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: "password": { 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "password", 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.982 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "password of InfluxDB server user", 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: "port": { 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "port", 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8086", 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.983 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server port", 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "false", 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T00:06:43.984 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: "threads": { 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threads", 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "32", 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T00:06:43.985 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: "username": { 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "username", 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "username of InfluxDB server user", 2024-03-18T00:06:43.986 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.987 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.987 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.987 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.987 INFO:teuthology.orchestra.run.smithi012.stdout: "verify_ssl": { 2024-03-18T00:06:43.987 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "verify_ssl", 2024-03-18T00:06:43.987 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.987 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.987 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:43.987 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "true", 2024-03-18T00:06:43.987 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.987 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.987 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.987 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "insights", 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:43.988 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.989 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.990 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.991 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.991 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.991 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:43.991 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:43.991 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.991 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.991 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.991 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:43.991 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.991 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.991 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.991 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:43.992 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.993 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "iostat", 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.994 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.995 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:43.996 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.997 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:43.998 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "k8sevents", 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_event_retention_days": { 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ceph_event_retention_days", 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:43.999 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7", 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "config_check_secs": { 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_check_secs", 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.000 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.001 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "10", 2024-03-18T00:06:44.001 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.001 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.001 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T00:06:44.001 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.001 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.001 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.001 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.001 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.001 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.001 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.001 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.001 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.002 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.002 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.002 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.002 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.002 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.002 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.002 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.002 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.002 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.002 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.002 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.002 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.002 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.003 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.004 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.005 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.006 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.006 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.006 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.006 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.006 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.006 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.006 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.006 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.006 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.006 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.006 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.006 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.006 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.007 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "localpool", 2024-03-18T00:06:44.007 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.007 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.007 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.007 INFO:teuthology.orchestra.run.smithi012.stdout: "failure_domain": { 2024-03-18T00:06:44.007 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "failure_domain", 2024-03-18T00:06:44.007 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.007 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.007 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.007 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "host", 2024-03-18T00:06:44.007 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.007 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.007 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "failure domain for any created local pool", 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.008 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.009 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.010 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.011 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.012 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: "min_size": { 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_size", 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.013 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default min_size for any created local pool", 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: "num_rep": { 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "num_rep", 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3", 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.014 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default replica count for any created local pool", 2024-03-18T00:06:44.015 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.015 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.015 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.015 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_num": { 2024-03-18T00:06:44.015 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_num", 2024-03-18T00:06:44.015 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.015 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.015 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.015 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "128", 2024-03-18T00:06:44.015 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.015 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "prefix": { 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prefix", 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.016 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name prefix for any created local pool", 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: "subtree": { 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "subtree", 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "rack", 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.017 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.018 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T00:06:44.018 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T00:06:44.018 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.018 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.018 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.018 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.018 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.018 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.018 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mds_autoscaler", 2024-03-18T00:06:44.018 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.018 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.018 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.018 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.019 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.020 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.021 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.022 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.023 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirroring", 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.024 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.025 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.026 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.027 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.028 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.029 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.029 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.029 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.029 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.029 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.029 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.029 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.029 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.029 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.029 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "nfs", 2024-03-18T00:06:44.029 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.030 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.031 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.031 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.031 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.031 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.031 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.031 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.031 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.031 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.031 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.031 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.031 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.031 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.032 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.032 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.032 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.032 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.032 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.032 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.032 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.032 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.032 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.032 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.032 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.032 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.032 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.033 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.033 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.033 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.033 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.033 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.033 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.033 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.033 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.033 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.033 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.033 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.033 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.033 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.034 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-03-18T00:06:44.035 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "fail_fs": { 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "fail_fs", 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.036 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.038 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.039 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.040 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.041 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.042 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.042 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator": { 2024-03-18T00:06:44.042 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-03-18T00:06:44.042 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.042 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.042 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.042 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.042 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.042 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.042 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.042 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-03-18T00:06:44.042 INFO:teuthology.orchestra.run.smithi012.stdout: "rook", 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: "test_orchestrator" 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Orchestrator backend", 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_perf_query", 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.043 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.044 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.045 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.046 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.046 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.046 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.046 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.046 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.046 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.046 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.046 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.046 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.046 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.046 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.046 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.046 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.047 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.048 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.048 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.048 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.048 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.048 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.048 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.048 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.048 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.048 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.048 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.048 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.048 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.048 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.049 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.049 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.049 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.049 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.049 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_support", 2024-03-18T00:06:44.049 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.049 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.049 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.049 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.049 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.049 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.049 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.049 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.050 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.051 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.051 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.054 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.054 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.054 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.054 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.054 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.054 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.054 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.054 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.054 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.054 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.054 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.054 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.055 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.056 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_autoscaler", 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.057 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.058 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.059 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.060 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.060 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.060 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.060 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.060 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.060 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.060 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.060 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.060 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.060 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.060 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.060 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.061 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.061 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.061 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.061 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.061 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.061 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.061 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.061 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.061 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.061 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.061 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.062 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.062 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.062 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.062 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.062 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.062 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.062 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.062 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.062 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.062 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.062 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.062 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.063 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.063 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.063 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.063 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.063 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.063 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:44.063 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:44.063 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.063 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.063 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.063 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:44.063 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: "threshold": { 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threshold", 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-03-18T00:06:44.064 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1.0", 2024-03-18T00:06:44.065 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.065 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.065 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "scaling threshold", 2024-03-18T00:06:44.065 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T00:06:44.065 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.065 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.065 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.065 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.065 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.065 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.065 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "progress", 2024-03-18T00:06:44.065 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.065 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.066 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.066 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_pg_recovery_event": { 2024-03-18T00:06:44.066 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_pg_recovery_event", 2024-03-18T00:06:44.066 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.066 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.066 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.066 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.066 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.066 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.066 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.066 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T00:06:44.066 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.067 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.068 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.068 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.068 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.068 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.068 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.068 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.068 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.068 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.068 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.068 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.068 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.068 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.069 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.070 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.070 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.070 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.070 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.070 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.070 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.070 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.070 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.070 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.070 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.070 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.070 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.071 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.072 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.072 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.072 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.072 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.072 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.072 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.072 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.072 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.072 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.072 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.072 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.072 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: "max_completed_events": { 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_completed_events", 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "50", 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.073 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.074 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.074 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "number of past completed events to remember", 2024-03-18T00:06:44.074 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.074 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.074 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.074 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.074 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:44.074 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:44.074 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.074 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.074 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.074 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-18T00:06:44.074 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.075 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.075 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.075 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long the module is going to sleep", 2024-03-18T00:06:44.075 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.075 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.075 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.075 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.075 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.075 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.075 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.075 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus", 2024-03-18T00:06:44.075 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.075 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "cache": { 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cache", 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.076 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: "exclude_perf_counters": { 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "exclude_perf_counters", 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.077 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.078 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.079 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.080 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.081 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools": { 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools", 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "300", 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.084 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_interval": { 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_interval", 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15.0", 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "9283", 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: "stale_cache_strategy": { 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stale_cache_strategy", 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "log", 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "default", 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: "default", 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: "error" 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.089 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.090 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.091 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.092 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.093 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.094 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.095 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.096 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.096 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.096 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.096 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.096 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.096 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.096 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.096 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.096 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.096 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_snap_create": { 2024-03-18T00:06:44.096 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_snap_create", 2024-03-18T00:06:44.096 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.096 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: "mirror_snapshot_schedule": { 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.097 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.098 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.098 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.098 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.098 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.098 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.098 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.098 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.098 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.098 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.098 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.098 INFO:teuthology.orchestra.run.smithi012.stdout: "trash_purge_schedule": { 2024-03-18T00:06:44.098 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "trash_purge_schedule", 2024-03-18T00:06:44.098 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.099 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "restful", 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_auth": { 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_auth", 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.100 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.101 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.102 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.102 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.102 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.102 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.102 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.102 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.102 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.103 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.103 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.103 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.103 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.103 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.103 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.103 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.103 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.103 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.103 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.103 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.103 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.103 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.104 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.104 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.104 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.104 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.104 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.104 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.104 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.104 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.104 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.104 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.104 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.104 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.104 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.105 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.106 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.106 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.106 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.106 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.106 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.106 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.106 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.106 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.106 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.106 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.106 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.106 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.106 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.107 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.108 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rgw", 2024-03-18T00:06:44.109 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.110 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.111 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.112 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.113 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.114 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.115 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.115 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.115 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.115 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.115 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rook", 2024-03-18T00:06:44.115 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.115 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.115 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.115 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.115 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.115 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.115 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.115 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.116 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.117 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.118 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.118 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.118 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.118 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.118 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.118 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.118 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.118 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.118 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.118 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.118 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.118 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.119 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.120 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.120 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.120 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.120 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.120 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.120 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.120 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.120 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.120 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.120 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.120 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.120 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_class": { 2024-03-18T00:06:44.120 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "storage_class", 2024-03-18T00:06:44.121 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.121 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.121 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.121 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "local", 2024-03-18T00:06:44.121 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.121 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.121 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.121 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T00:06:44.121 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.121 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.121 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.121 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.122 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.122 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.122 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.122 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "selftest", 2024-03-18T00:06:44.122 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.122 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.122 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.122 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.122 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.122 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.122 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.122 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.122 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.123 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.124 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.126 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.126 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.126 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.126 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.126 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.126 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.126 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.126 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.126 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.126 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.126 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: "roption1": { 2024-03-18T00:06:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption1", 2024-03-18T00:06:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: "roption2": { 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption2", 2024-03-18T00:06:44.128 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "xyz", 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption1": { 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption1", 2024-03-18T00:06:44.129 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption2": { 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption2", 2024-03-18T00:06:44.130 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.131 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.131 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.131 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.131 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.131 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.131 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.131 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.131 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.131 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.131 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.131 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.131 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption3": { 2024-03-18T00:06:44.131 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption3", 2024-03-18T00:06:44.132 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.132 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.132 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.132 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.132 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.132 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.132 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.132 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.132 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.132 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.132 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.132 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.132 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption4": { 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption4", 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.133 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption5": { 2024-03-18T00:06:44.134 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption5", 2024-03-18T00:06:44.134 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.134 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.134 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.134 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.134 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.134 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.134 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.134 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.134 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.134 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.134 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.134 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption6": { 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption6", 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.135 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.136 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption7": { 2024-03-18T00:06:44.136 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption7", 2024-03-18T00:06:44.136 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.136 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.136 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.136 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.136 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-18T00:06:44.136 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "42", 2024-03-18T00:06:44.136 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.136 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.136 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.136 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.136 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.137 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.137 INFO:teuthology.orchestra.run.smithi012.stdout: "testkey": { 2024-03-18T00:06:44.137 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testkey", 2024-03-18T00:06:44.137 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.137 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.137 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.137 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.137 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.137 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.137 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.137 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.137 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.137 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.138 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.138 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.138 INFO:teuthology.orchestra.run.smithi012.stdout: "testlkey": { 2024-03-18T00:06:44.138 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testlkey", 2024-03-18T00:06:44.138 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.138 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.138 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.138 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.138 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.138 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.138 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.138 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.138 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "testnewline": { 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testnewline", 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.139 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_m_granularity": { 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_m_granularity", 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.140 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: "dump_on_update": { 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dump_on_update", 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.141 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "dump database to debug log on update", 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.142 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.143 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.144 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.145 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.146 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stats", 2024-03-18T00:06:44.147 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.148 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.149 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.150 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.150 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.150 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.150 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.150 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.150 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.150 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.150 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.150 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.150 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.150 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.150 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.150 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.151 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.152 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "status", 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.153 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.154 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.155 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.156 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.157 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telegraf", 2024-03-18T00:06:44.158 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "address": { 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "address", 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.159 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15", 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.160 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.161 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.162 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.163 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.164 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.165 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telemetry", 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_basic": { 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_basic", 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.166 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_crash": { 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_crash", 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.167 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_device": { 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_device", 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.168 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.169 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.169 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.169 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T00:06:44.169 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.169 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.169 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.169 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.169 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_ident": { 2024-03-18T00:06:44.169 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_ident", 2024-03-18T00:06:44.169 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.169 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.169 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.170 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.170 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.170 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.170 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.170 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T00:06:44.170 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.170 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.170 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.170 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.170 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_perf": { 2024-03-18T00:06:44.170 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_perf", 2024-03-18T00:06:44.170 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.170 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "contact": { 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "contact", 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.171 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: "description": { 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "description", 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.172 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.173 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.173 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.173 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.173 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.173 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.173 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.173 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.173 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.173 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.173 INFO:teuthology.orchestra.run.smithi012.stdout: "device_url": { 2024-03-18T00:06:44.173 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_url", 2024-03-18T00:06:44.173 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.173 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.174 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.175 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "24", 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "8", 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "last_opt_revision": { 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "last_opt_revision", 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.176 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1", 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard": { 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard", 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.177 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard_description": { 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard_description", 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.178 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.181 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.182 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.183 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: "organization": { 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "organization", 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.184 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "proxy": { 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "proxy", 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.185 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.186 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.186 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.186 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.186 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.186 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.186 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.186 INFO:teuthology.orchestra.run.smithi012.stdout: "url": { 2024-03-18T00:06:44.186 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url", 2024-03-18T00:06:44.186 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.186 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.186 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.186 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test_orchestrator", 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.187 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.188 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.189 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.190 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.191 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.191 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.191 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.191 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.191 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.191 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.191 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.191 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.191 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.191 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.191 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.191 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.191 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.192 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.193 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.194 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.194 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.194 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.194 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.194 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.194 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.194 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.194 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.194 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.194 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.194 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.194 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.194 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.195 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.196 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.197 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_clones": { 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_clones", 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.198 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4", 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: "periodic_async_work": { 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "periodic_async_work", 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.199 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Periodically check for async work", 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_delay": { 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_delay", 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.200 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_no_wait": { 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.201 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.202 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix", 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "discovery_interval": { 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "discovery_interval", 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "100", 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.203 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: "identifier": { 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "identifier", 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.204 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.205 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.206 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.206 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.206 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.206 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.206 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.206 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.206 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.206 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.206 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.206 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.206 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.206 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.206 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.207 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.208 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.209 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.210 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_host": { 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_host", 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.212 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_port": { 2024-03-18T00:06:44.212 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_port", 2024-03-18T00:06:44.212 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.212 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.212 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.212 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10051", 2024-03-18T00:06:44.212 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.212 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.212 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.212 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.212 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_sender": { 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_sender", 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.213 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.214 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.214 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.214 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.214 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.214 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.214 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-18T00:06:44.214 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.214 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.214 INFO:teuthology.orchestra.run.smithi012.stdout: "gid": 24113, 2024-03-18T00:06:44.214 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "y", 2024-03-18T00:06:44.214 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr_features": 4540701547738038271, 2024-03-18T00:06:44.214 INFO:teuthology.orchestra.run.smithi012.stdout: "available_modules": [ 2024-03-18T00:06:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "alerts", 2024-03-18T00:06:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.215 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.216 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.217 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.218 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.218 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.218 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.218 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.218 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.218 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.218 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.218 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.218 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.218 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.218 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.218 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.218 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.219 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.219 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.219 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.219 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.219 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.219 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.219 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.219 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.219 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.219 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.219 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.219 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.219 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.220 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_destination": { 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_destination", 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.221 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.222 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.222 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.222 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.222 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email address to send alerts to", 2024-03-18T00:06:44.222 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.222 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.222 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.222 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_from_name": { 2024-03-18T00:06:44.222 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_from_name", 2024-03-18T00:06:44.222 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.222 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.222 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "Ceph", 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email From: name", 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_host": { 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_host", 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.223 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP server", 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_password": { 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_password", 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.224 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Password to authenticate with", 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_port": { 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_port", 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.225 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "465", 2024-03-18T00:06:44.226 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.226 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.226 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.226 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP port", 2024-03-18T00:06:44.226 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.226 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.226 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.226 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_sender": { 2024-03-18T00:06:44.226 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_sender", 2024-03-18T00:06:44.226 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.226 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.226 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP envelope sender", 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_ssl": { 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_ssl", 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.227 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.228 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.228 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.228 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.228 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.228 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T00:06:44.228 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.228 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.228 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.228 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_user": { 2024-03-18T00:06:44.228 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_user", 2024-03-18T00:06:44.228 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.228 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.229 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.229 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.229 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.229 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.229 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.229 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "User to authenticate as", 2024-03-18T00:06:44.229 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.229 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.229 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.229 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.229 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.229 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "balancer", 2024-03-18T00:06:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: "active": { 2024-03-18T00:06:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "active", 2024-03-18T00:06:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_time": { 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_time", 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0000", 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.231 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_weekday": { 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_weekday", 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-03-18T00:06:44.232 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_max_iterations": { 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_max_iterations", 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "25", 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "250", 2024-03-18T00:06:44.233 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_metrics": { 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_metrics", 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.234 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T00:06:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T00:06:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_step": { 2024-03-18T00:06:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_step", 2024-03-18T00:06:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.5", 2024-03-18T00:06:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0.001", 2024-03-18T00:06:44.235 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "0.999", 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "aggressiveness of optimization", 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "end_time": { 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_time", 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2359", 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.236 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T00:06:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T00:06:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "end_weekday": { 2024-03-18T00:06:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_weekday", 2024-03-18T00:06:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-03-18T00:06:44.237 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-03-18T00:06:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T00:06:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T00:06:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.238 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.239 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.242 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "min_score": { 2024-03-18T00:06:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_score", 2024-03-18T00:06:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.243 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.244 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "upmap", 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "crush-compat", 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "none", 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "read", 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap", 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap-read" 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Balancer mode", 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.245 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_ids": { 2024-03-18T00:06:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_ids", 2024-03-18T00:06:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T00:06:44.246 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T00:06:44.247 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_deviation": { 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_deviation", 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T00:06:44.248 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_optimizations": { 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_optimizations", 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.249 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm", 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_down_multiplier": { 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_down_multiplier", 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.250 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_refresh_rate": { 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_refresh_rate", 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.251 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "20", 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_starting_port": { 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_starting_port", 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.252 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4721", 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_ptrace": { 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_ptrace", 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.253 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T00:06:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T00:06:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_interval": { 2024-03-18T00:06:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_interval", 2024-03-18T00:06:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.254 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:44.255 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T00:06:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_memory_target_ratio": { 2024-03-18T00:06:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T00:06:44.258 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.7", 2024-03-18T00:06:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T00:06:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm_log_destination": { 2024-03-18T00:06:44.259 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm_log_destination", 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "file", 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "file,syslog", 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "syslog" 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.260 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "cgroups_split": { 2024-03-18T00:06:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cgroups_split", 2024-03-18T00:06:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T00:06:44.261 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "config_checks_enabled": { 2024-03-18T00:06:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_checks_enabled", 2024-03-18T00:06:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.262 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T00:06:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "config_dashboard": { 2024-03-18T00:06:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_dashboard", 2024-03-18T00:06:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.263 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_alertmanager": { 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_alertmanager", 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.264 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_base": { 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_base", 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.265 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Container image name, without the tag", 2024-03-18T00:06:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_elasticsearch": { 2024-03-18T00:06:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_elasticsearch", 2024-03-18T00:06:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T00:06:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.266 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "elasticsearch container image", 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_grafana": { 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_grafana", 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.267 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_haproxy": { 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_haproxy", 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.268 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "HAproxy container image", 2024-03-18T00:06:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_agent": { 2024-03-18T00:06:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_agent", 2024-03-18T00:06:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T00:06:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.269 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger agent container image", 2024-03-18T00:06:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_collector": { 2024-03-18T00:06:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_collector", 2024-03-18T00:06:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T00:06:44.270 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger collector container image", 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_query": { 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_query", 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.271 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger query container image", 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_keepalived": { 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_keepalived", 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.272 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Keepalived container image", 2024-03-18T00:06:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_loki": { 2024-03-18T00:06:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_loki", 2024-03-18T00:06:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T00:06:44.273 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Loki container image", 2024-03-18T00:06:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_node_exporter": { 2024-03-18T00:06:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_node_exporter", 2024-03-18T00:06:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.274 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_nvmeof": { 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_nvmeof", 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.275 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Nvme-of container image", 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_prometheus": { 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_prometheus", 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T00:06:44.276 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_promtail": { 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_promtail", 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T00:06:44.277 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Promtail container image", 2024-03-18T00:06:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_snmp_gateway": { 2024-03-18T00:06:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_snmp_gateway", 2024-03-18T00:06:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.278 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SNMP Gateway container image", 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "container_init": { 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_init", 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.279 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "daemon_cache_timeout": { 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "daemon_cache_timeout", 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "default_cephadm_command_timeout": { 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "900", 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.281 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T00:06:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "default_registry": { 2024-03-18T00:06:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_registry", 2024-03-18T00:06:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io", 2024-03-18T00:06:44.282 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "device_cache_timeout": { 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_cache_timeout", 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1800", 2024-03-18T00:06:44.283 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache device inventory", 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "device_enhanced_scan": { 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_enhanced_scan", 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.284 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "facts_cache_timeout": { 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "facts_cache_timeout", 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:44.285 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache host facts data", 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "host_check_interval": { 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "host_check_interval", 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:44.286 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to perform a host check", 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "hw_monitoring": { 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hw_monitoring", 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.287 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "inventory_list_all": { 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "inventory_list_all", 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.288 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.289 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "log_refresh_metadata": { 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_refresh_metadata", 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.293 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.294 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T00:06:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T00:06:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.295 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.296 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "*", 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "max_count_per_host": { 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_count_per_host", 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.297 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of daemons per service per host", 2024-03-18T00:06:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "max_osd_draining_count": { 2024-03-18T00:06:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_osd_draining_count", 2024-03-18T00:06:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.298 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "migration_current": { 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "migration_current", 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.299 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "internal - do not modify", 2024-03-18T00:06:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-03-18T00:06:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-03-18T00:06:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.300 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "root", 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm-package", 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "root" 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "oob_default_addr": { 2024-03-18T00:06:44.301 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "oob_default_addr", 2024-03-18T00:06:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "169.254.1.1", 2024-03-18T00:06:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T00:06:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.302 INFO:teuthology.orchestra.run.smithi012.stdout: "prometheus_alerts_path": { 2024-03-18T00:06:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus_alerts_path", 2024-03-18T00:06:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T00:06:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T00:06:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_insecure": { 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_insecure", 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.304 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_password": { 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_password", 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.305 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_url": { 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_url", 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.306 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_username": { 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_username", 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.307 INFO:teuthology.orchestra.run.smithi012.stdout: "secure_monitoring_stack": { 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "secure_monitoring_stack", 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.308 INFO:teuthology.orchestra.run.smithi012.stdout: "service_discovery_port": { 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "service_discovery_port", 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8765", 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "cephadm service discovery port", 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.309 INFO:teuthology.orchestra.run.smithi012.stdout: "ssh_config_file": { 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssh_config_file", 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.310 INFO:teuthology.orchestra.run.smithi012.stdout: "use_agent": { 2024-03-18T00:06:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_agent", 2024-03-18T00:06:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T00:06:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "use_repo_digest": { 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_repo_digest", 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.312 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_failed_host_check": { 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_failed_host_check", 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.313 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_daemons": { 2024-03-18T00:06:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_daemons", 2024-03-18T00:06:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T00:06:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_hosts": { 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_hosts", 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.315 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crash", 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.316 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.317 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.318 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.319 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.320 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "retain_interval": { 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retain_interval", 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "31536000", 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.321 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_recent_interval": { 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_recent_interval", 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1209600", 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.322 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dashboard", 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.323 INFO:teuthology.orchestra.run.smithi012.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.324 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.325 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.326 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_ENABLED": { 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T00:06:44.327 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T00:06:44.328 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T00:06:44.329 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T00:06:44.330 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T00:06:44.331 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T00:06:44.332 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T00:06:44.333 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.334 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.334 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.334 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.334 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.334 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.334 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.334 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.334 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.334 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.334 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.334 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T00:06:44.334 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T00:06:44.335 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T00:06:44.336 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.337 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.338 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.338 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-18T00:06:44.338 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.338 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.338 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.338 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.338 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.338 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.338 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.338 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T00:06:44.338 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T00:06:44.338 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.339 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.339 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.339 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.339 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.339 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.339 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.339 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.339 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.339 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.339 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.339 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_URL": { 2024-03-18T00:06:44.339 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_URL", 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T00:06:44.340 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T00:06:44.341 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T00:06:44.342 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.343 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.344 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.345 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.346 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "{}", 2024-03-18T00:06:44.346 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.346 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.346 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.346 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.346 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.346 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.346 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.346 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T00:06:44.346 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T00:06:44.346 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.346 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.347 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.348 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.349 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.350 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.350 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.350 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.350 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.350 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.350 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.350 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.350 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.350 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T00:06:44.350 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T00:06:44.350 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.350 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.351 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.352 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.353 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.354 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.355 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.356 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.357 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.358 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.358 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.358 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.358 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.358 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.358 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.358 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T00:06:44.358 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T00:06:44.358 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.358 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.358 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.358 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8", 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.359 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "45", 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.360 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-18T00:06:44.361 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.362 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.363 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.363 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.363 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.363 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.363 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.363 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.363 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T00:06:44.363 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T00:06:44.363 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.363 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.363 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.364 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.365 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.366 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.367 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: "cross_origin_url": { 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cross_origin_url", 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.368 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "crt_file": { 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crt_file", 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.369 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: "debug": { 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "debug", 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.370 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable/disable debug options", 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: "jwt_token_ttl": { 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "jwt_token_ttl", 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "28800", 2024-03-18T00:06:44.371 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.372 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.373 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.374 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.374 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.374 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.374 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.374 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.374 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.374 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.374 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.374 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.374 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.374 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.374 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.374 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.376 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.377 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: "motd": { 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "motd", 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.378 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "The message of the day", 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_resolve_ip_addr": { 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.379 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-03-18T00:06:44.380 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8080", 2024-03-18T00:06:44.381 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.382 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl_server_port": { 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl_server_port", 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8443", 2024-03-18T00:06:44.383 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "redirect", 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.384 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.385 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.385 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.385 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect" 2024-03-18T00:06:44.385 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.385 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.385 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.385 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.385 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.385 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.385 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-03-18T00:06:44.385 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-03-18T00:06:44.385 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.385 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.386 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.386 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-03-18T00:06:44.386 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-03-18T00:06:44.386 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-03-18T00:06:44.386 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.386 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.386 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.386 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.386 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.386 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.386 INFO:teuthology.orchestra.run.smithi012.stdout: "url_prefix": { 2024-03-18T00:06:44.386 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url_prefix", 2024-03-18T00:06:44.386 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.387 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-03-18T00:06:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_monitoring": { 2024-03-18T00:06:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_monitoring", 2024-03-18T00:06:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.388 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "monitor device health metrics", 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.389 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.390 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.391 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.392 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.393 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: "mark_out_threshold": { 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mark_out_threshold", 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2419200", 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.394 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_name": { 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_name", 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "device_health_metrics", 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.395 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: "retention_period": { 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retention_period", 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15552000", 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.396 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain device health metrics", 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_frequency": { 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_frequency", 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.397 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: "self_heal": { 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "self_heal", 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.398 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T00:06:44.399 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.399 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.399 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.399 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.399 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:44.399 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:44.399 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.399 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.399 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.399 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:44.399 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.399 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.399 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.400 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T00:06:44.400 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.400 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.400 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.400 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.400 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_threshold": { 2024-03-18T00:06:44.400 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_threshold", 2024-03-18T00:06:44.400 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.400 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.400 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.400 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7257600", 2024-03-18T00:06:44.400 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.400 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "diskprediction_local", 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.401 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.402 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.403 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.404 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.405 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.406 INFO:teuthology.orchestra.run.smithi012.stdout: "predict_interval": { 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predict_interval", 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.407 INFO:teuthology.orchestra.run.smithi012.stdout: "predictor_model": { 2024-03-18T00:06:44.408 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predictor_model", 2024-03-18T00:06:44.408 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.408 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.408 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.408 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "prophetstor", 2024-03-18T00:06:44.408 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.408 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.408 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.408 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.408 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.408 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.408 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.408 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.409 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "influx", 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": false, 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "influxdb python module not found", 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: "batch_size": { 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "batch_size", 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5000", 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.410 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: "database": { 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "database", 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "ceph", 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.411 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": { 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hostname", 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.412 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server hostname", 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "30", 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "5", 2024-03-18T00:06:44.413 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.414 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.414 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T00:06:44.414 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.414 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.414 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.414 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.414 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.414 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.414 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.414 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.414 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.414 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.414 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.415 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.417 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.418 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: "password": { 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "password", 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.419 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "password of InfluxDB server user", 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "port": { 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "port", 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8086", 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.420 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server port", 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "false", 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.421 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.422 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T00:06:44.422 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.422 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.422 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.422 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.422 INFO:teuthology.orchestra.run.smithi012.stdout: "threads": { 2024-03-18T00:06:44.422 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threads", 2024-03-18T00:06:44.422 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.422 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.422 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.422 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-18T00:06:44.422 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-18T00:06:44.422 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "32", 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "username": { 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "username", 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.423 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "username of InfluxDB server user", 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "verify_ssl": { 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "verify_ssl", 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "true", 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.424 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T00:06:44.425 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.425 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.425 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.425 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.425 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.425 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.425 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.425 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "insights", 2024-03-18T00:06:44.425 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.425 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.425 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.425 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.425 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.426 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.427 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.428 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.429 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.430 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "iostat", 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.431 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.432 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.432 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.432 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.432 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.432 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.432 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.432 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.432 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.432 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.432 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.432 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.432 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.432 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.433 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.434 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.435 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.436 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.436 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.436 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.436 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.436 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.436 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.436 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.436 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.436 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "k8sevents", 2024-03-18T00:06:44.436 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.436 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.436 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.436 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_event_retention_days": { 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ceph_event_retention_days", 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7", 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "config_check_secs": { 2024-03-18T00:06:44.437 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_check_secs", 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "10", 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.438 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.439 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.439 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.439 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.439 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.439 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.439 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.439 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.439 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.439 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.439 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.439 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.439 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.439 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.440 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.440 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.440 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.440 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.440 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.440 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.440 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.440 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.440 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.440 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.440 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.440 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.440 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.441 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.442 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.443 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "localpool", 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: "failure_domain": { 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "failure_domain", 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "host", 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.444 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "failure domain for any created local pool", 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.445 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.446 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.446 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.446 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.446 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.446 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.446 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.446 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.446 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.446 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.446 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.446 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.446 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.447 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.448 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.448 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.448 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.448 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.448 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.448 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.448 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.448 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.448 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.448 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.448 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.448 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.448 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.449 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.449 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.449 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.449 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.449 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.449 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.449 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.449 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.449 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.449 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.449 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.449 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.449 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: "min_size": { 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_size", 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.450 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default min_size for any created local pool", 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: "num_rep": { 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "num_rep", 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.451 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3", 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default replica count for any created local pool", 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_num": { 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_num", 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.452 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "128", 2024-03-18T00:06:44.453 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.453 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.453 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.453 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T00:06:44.453 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.453 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.453 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.453 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.453 INFO:teuthology.orchestra.run.smithi012.stdout: "prefix": { 2024-03-18T00:06:44.454 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prefix", 2024-03-18T00:06:44.454 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.454 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.454 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.454 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.454 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.454 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.454 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.454 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name prefix for any created local pool", 2024-03-18T00:06:44.454 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.454 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.454 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.454 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: "subtree": { 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "subtree", 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "rack", 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.455 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mds_autoscaler", 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.456 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.457 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.458 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.458 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.458 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.458 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.458 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.458 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.458 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.458 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.458 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.458 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.461 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.462 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.462 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.462 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.462 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.462 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.462 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.462 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.462 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.462 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.462 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.462 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.462 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.463 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.463 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.463 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.463 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.463 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.463 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.463 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.463 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.463 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.463 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.463 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.463 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.463 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.464 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.465 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirroring", 2024-03-18T00:06:44.465 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.465 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.465 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.465 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.465 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.465 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.465 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.465 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.465 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.465 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.465 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.465 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.466 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.467 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.468 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.469 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "nfs", 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.470 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.471 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.471 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.471 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.471 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.471 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.471 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.471 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.471 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.471 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.471 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.471 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.471 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.471 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.472 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.473 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.474 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "fail_fs": { 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "fail_fs", 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.476 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.477 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.478 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.479 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.481 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator": { 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: "rook", 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: "test_orchestrator" 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Orchestrator backend", 2024-03-18T00:06:44.482 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_perf_query", 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.483 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.484 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.485 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.486 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.487 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_support", 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.488 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.489 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.490 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.491 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.492 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.493 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_autoscaler", 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.494 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.495 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.496 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.497 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.498 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.498 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.498 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.498 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.498 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.498 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.498 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.498 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.498 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.498 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.498 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.498 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.498 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.499 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: "threshold": { 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threshold", 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1.0", 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "scaling threshold", 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T00:06:44.500 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "progress", 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_pg_recovery_event": { 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_pg_recovery_event", 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.501 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.502 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.503 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.504 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.505 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.506 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.507 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "max_completed_events": { 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_completed_events", 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "50", 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.508 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "number of past completed events to remember", 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.509 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long the module is going to sleep", 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus", 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: "cache": { 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cache", 2024-03-18T00:06:44.510 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: "exclude_perf_counters": { 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "exclude_perf_counters", 2024-03-18T00:06:44.511 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.512 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.513 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.514 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.515 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.516 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools": { 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools", 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.517 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.518 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "300", 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_interval": { 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_interval", 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15.0", 2024-03-18T00:06:44.519 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.520 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "9283", 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.521 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: "stale_cache_strategy": { 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stale_cache_strategy", 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "log", 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.522 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.523 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.523 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.523 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.523 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.523 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.523 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-03-18T00:06:44.523 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-03-18T00:06:44.523 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.523 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.523 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.523 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "default", 2024-03-18T00:06:44.523 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.523 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: "default", 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: "error" 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.524 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.525 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.526 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.527 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.528 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.529 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.530 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_snap_create": { 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_snap_create", 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.531 INFO:teuthology.orchestra.run.smithi012.stdout: "mirror_snapshot_schedule": { 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "trash_purge_schedule": { 2024-03-18T00:06:44.532 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "trash_purge_schedule", 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.533 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "restful", 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_auth": { 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_auth", 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.534 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.535 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.536 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.537 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.538 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.539 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.540 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.541 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-18T00:06:44.542 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-18T00:06:44.542 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.542 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.542 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.542 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.542 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.542 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.542 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.542 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.542 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.542 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.542 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.542 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rgw", 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.543 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.544 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.545 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.546 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.547 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rook", 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.550 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.551 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.551 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.551 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.551 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.551 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.551 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.551 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.551 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.551 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.551 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.551 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.551 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.551 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.552 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_class": { 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "storage_class", 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.553 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "local", 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "selftest", 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.554 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.555 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.556 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.557 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.558 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "roption1": { 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption1", 2024-03-18T00:06:44.559 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "roption2": { 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption2", 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.560 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "xyz", 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption1": { 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption1", 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.561 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption2": { 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption2", 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.562 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption3": { 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption3", 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.563 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption4": { 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption4", 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.564 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption5": { 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption5", 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.565 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption6": { 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption6", 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.566 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption7": { 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption7", 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "42", 2024-03-18T00:06:44.567 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "testkey": { 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testkey", 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.568 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: "testlkey": { 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testlkey", 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.569 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "testnewline": { 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testnewline", 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.570 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_m_granularity": { 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_m_granularity", 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.571 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "dump_on_update": { 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dump_on_update", 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.572 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "dump database to debug log on update", 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.573 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.574 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.575 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.576 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.577 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stats", 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.578 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.579 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.580 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.581 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.582 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "status", 2024-03-18T00:06:44.583 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.585 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.586 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.587 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.588 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telegraf", 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "address": { 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "address", 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.589 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15", 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.590 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.591 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.592 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.593 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.594 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.595 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telemetry", 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_basic": { 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_basic", 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.596 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_crash": { 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_crash", 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.597 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_device": { 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_device", 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.598 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_ident": { 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_ident", 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.599 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_perf": { 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_perf", 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.600 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: "contact": { 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "contact", 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.601 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: "description": { 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "description", 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.602 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "device_url": { 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_url", 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.603 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.604 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.604 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.604 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.604 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.604 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-03-18T00:06:44.604 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-03-18T00:06:44.604 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.604 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.604 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.604 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "24", 2024-03-18T00:06:44.605 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "8", 2024-03-18T00:06:44.606 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.606 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.606 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.606 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.606 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.606 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.606 INFO:teuthology.orchestra.run.smithi012.stdout: "last_opt_revision": { 2024-03-18T00:06:44.606 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "last_opt_revision", 2024-03-18T00:06:44.606 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.606 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.606 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.606 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1", 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard": { 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard", 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.607 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard_description": { 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard_description", 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.608 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.609 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.610 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.611 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.612 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.613 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: "organization": { 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "organization", 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.614 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "proxy": { 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "proxy", 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.615 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: "url": { 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url", 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.616 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test_orchestrator", 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.617 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.618 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.618 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.618 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.618 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.618 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.618 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.618 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.618 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.618 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.618 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.618 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.618 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.618 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.619 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.620 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.621 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.622 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.622 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.622 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.622 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.622 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.622 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.622 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.622 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.622 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.622 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.622 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.622 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.622 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.623 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.624 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.625 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.625 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.625 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.625 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.625 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.625 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.625 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.625 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.625 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.625 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.625 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.625 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.625 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.626 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.626 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.626 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.626 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.626 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.626 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.626 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.626 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.626 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.626 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.627 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.627 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.627 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.627 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.627 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.627 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.627 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.627 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.627 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.627 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.627 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.627 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_clones": { 2024-03-18T00:06:44.628 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_clones", 2024-03-18T00:06:44.629 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.629 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.629 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.629 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4", 2024-03-18T00:06:44.629 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.629 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.629 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.629 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T00:06:44.629 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.629 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.630 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.630 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.630 INFO:teuthology.orchestra.run.smithi012.stdout: "periodic_async_work": { 2024-03-18T00:06:44.630 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "periodic_async_work", 2024-03-18T00:06:44.630 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.630 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.630 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.630 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.630 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.631 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.631 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.631 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Periodically check for async work", 2024-03-18T00:06:44.631 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.631 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.631 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.631 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.631 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_delay": { 2024-03-18T00:06:44.631 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_delay", 2024-03-18T00:06:44.631 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.632 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.632 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.632 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:44.632 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.632 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.632 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.632 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T00:06:44.632 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.632 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.632 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.632 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.632 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_no_wait": { 2024-03-18T00:06:44.633 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T00:06:44.633 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.633 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.633 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.633 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.633 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.633 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.633 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.633 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T00:06:44.633 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.633 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.633 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.633 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.634 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.634 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.634 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.634 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix", 2024-03-18T00:06:44.634 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.634 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.634 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.634 INFO:teuthology.orchestra.run.smithi012.stdout: "discovery_interval": { 2024-03-18T00:06:44.634 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "discovery_interval", 2024-03-18T00:06:44.634 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:44.634 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.634 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.634 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "100", 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: "identifier": { 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "identifier", 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.635 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.636 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.636 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.636 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.636 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.636 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.636 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.636 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.636 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:44.636 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:44.636 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.636 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.636 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.637 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.638 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.639 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.640 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.641 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_host": { 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_host", 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.642 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_port": { 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_port", 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10051", 2024-03-18T00:06:44.643 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_sender": { 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_sender", 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T00:06:44.644 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: "modules": [ 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-03-18T00:06:44.645 INFO:teuthology.orchestra.run.smithi012.stdout: "dashboard", 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: "iostat", 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: "nfs", 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: "restful" 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: "available_modules": [ 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "alerts", 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.646 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.647 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.648 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.648 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.648 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.648 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.648 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.648 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.648 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.648 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.648 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.648 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.648 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.648 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.649 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.650 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.650 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.650 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.650 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.650 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.650 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.650 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.650 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.650 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.650 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.650 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.650 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.650 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.651 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_destination": { 2024-03-18T00:06:44.652 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_destination", 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email address to send alerts to", 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_from_name": { 2024-03-18T00:06:44.653 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_from_name", 2024-03-18T00:06:44.654 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.654 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.654 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.654 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "Ceph", 2024-03-18T00:06:44.654 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.654 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.654 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.654 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email From: name", 2024-03-18T00:06:44.654 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.654 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.654 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.654 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.654 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_host": { 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_host", 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP server", 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.655 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_password": { 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_password", 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Password to authenticate with", 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.656 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_port": { 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_port", 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "465", 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP port", 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.657 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_sender": { 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_sender", 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP envelope sender", 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.658 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_ssl": { 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_ssl", 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_user": { 2024-03-18T00:06:44.659 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_user", 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "User to authenticate as", 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.660 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "balancer", 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "active": { 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "active", 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.661 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_time": { 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_time", 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0000", 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.662 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_weekday": { 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_weekday", 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.663 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T00:06:44.664 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T00:06:44.664 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.665 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.665 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.665 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_max_iterations": { 2024-03-18T00:06:44.665 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_max_iterations", 2024-03-18T00:06:44.665 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:44.665 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.665 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.665 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "25", 2024-03-18T00:06:44.665 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-18T00:06:44.665 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "250", 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_metrics": { 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_metrics", 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.666 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_step": { 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_step", 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.5", 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0.001", 2024-03-18T00:06:44.667 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "0.999", 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "aggressiveness of optimization", 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "end_time": { 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_time", 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2359", 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.668 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: "end_weekday": { 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_weekday", 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-03-18T00:06:44.669 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.670 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.671 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.671 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.671 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.671 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.671 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.671 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.671 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.671 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.671 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.671 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.671 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.671 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.672 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.672 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.672 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.672 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.672 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.672 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.672 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.672 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.672 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.672 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.672 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.672 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.672 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.673 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.674 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: "min_score": { 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_score", 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.675 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.676 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "upmap", 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "crush-compat", 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "none", 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "read", 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap", 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap-read" 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Balancer mode", 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.677 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_ids": { 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_ids", 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.678 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_deviation": { 2024-03-18T00:06:44.679 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_deviation", 2024-03-18T00:06:44.680 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.680 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.680 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.680 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-18T00:06:44.680 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-18T00:06:44.680 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.680 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.680 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T00:06:44.680 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T00:06:44.680 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.680 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.680 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.680 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_optimizations": { 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_optimizations", 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.681 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.682 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.682 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm", 2024-03-18T00:06:44.682 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.682 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.682 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.682 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_down_multiplier": { 2024-03-18T00:06:44.682 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_down_multiplier", 2024-03-18T00:06:44.682 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.682 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.682 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.682 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-03-18T00:06:44.682 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.682 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_refresh_rate": { 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_refresh_rate", 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "20", 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.683 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_starting_port": { 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_starting_port", 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4721", 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_ptrace": { 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_ptrace", 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.685 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_interval": { 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_interval", 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.686 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_memory_target_ratio": { 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.7", 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.687 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.688 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.688 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T00:06:44.688 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.688 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.688 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.688 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.688 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm_log_destination": { 2024-03-18T00:06:44.688 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm_log_destination", 2024-03-18T00:06:44.688 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.688 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.688 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.689 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.689 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.689 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.689 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.689 INFO:teuthology.orchestra.run.smithi012.stdout: "file", 2024-03-18T00:06:44.689 INFO:teuthology.orchestra.run.smithi012.stdout: "file,syslog", 2024-03-18T00:06:44.689 INFO:teuthology.orchestra.run.smithi012.stdout: "syslog" 2024-03-18T00:06:44.689 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.689 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T00:06:44.690 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.690 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.690 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.690 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.690 INFO:teuthology.orchestra.run.smithi012.stdout: "cgroups_split": { 2024-03-18T00:06:44.690 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cgroups_split", 2024-03-18T00:06:44.690 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.690 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.690 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.691 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.691 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.691 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.691 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.691 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T00:06:44.691 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.691 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.691 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.691 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: "config_checks_enabled": { 2024-03-18T00:06:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_checks_enabled", 2024-03-18T00:06:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T00:06:44.692 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: "config_dashboard": { 2024-03-18T00:06:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_dashboard", 2024-03-18T00:06:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.693 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_alertmanager": { 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_alertmanager", 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.694 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_base": { 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_base", 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.695 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Container image name, without the tag", 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_elasticsearch": { 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_elasticsearch", 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.696 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "elasticsearch container image", 2024-03-18T00:06:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_grafana": { 2024-03-18T00:06:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_grafana", 2024-03-18T00:06:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T00:06:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.697 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_haproxy": { 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_haproxy", 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.698 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "HAproxy container image", 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_agent": { 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_agent", 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.699 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger agent container image", 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_collector": { 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_collector", 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.700 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.701 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger collector container image", 2024-03-18T00:06:44.701 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.701 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.701 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.701 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_query": { 2024-03-18T00:06:44.701 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_query", 2024-03-18T00:06:44.701 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.701 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.701 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.701 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T00:06:44.701 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.701 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger query container image", 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_keepalived": { 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_keepalived", 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.702 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.703 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.703 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Keepalived container image", 2024-03-18T00:06:44.703 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.703 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.703 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.703 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.703 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_loki": { 2024-03-18T00:06:44.703 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_loki", 2024-03-18T00:06:44.703 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.703 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.703 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.703 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T00:06:44.703 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Loki container image", 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_node_exporter": { 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_node_exporter", 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T00:06:44.704 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_nvmeof": { 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_nvmeof", 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T00:06:44.705 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.706 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.706 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.706 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Nvme-of container image", 2024-03-18T00:06:44.706 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.706 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.706 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.706 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.706 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_prometheus": { 2024-03-18T00:06:44.707 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_prometheus", 2024-03-18T00:06:44.707 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.707 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.707 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.707 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T00:06:44.707 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.707 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.707 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.708 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-03-18T00:06:44.708 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.708 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.708 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.708 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.708 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_promtail": { 2024-03-18T00:06:44.708 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_promtail", 2024-03-18T00:06:44.708 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.708 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.709 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.709 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T00:06:44.709 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.709 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.709 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.709 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Promtail container image", 2024-03-18T00:06:44.709 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.709 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.710 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.710 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.710 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_snmp_gateway": { 2024-03-18T00:06:44.710 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_snmp_gateway", 2024-03-18T00:06:44.710 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.710 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.710 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.710 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T00:06:44.710 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.710 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.710 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.710 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SNMP Gateway container image", 2024-03-18T00:06:44.710 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "container_init": { 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_init", 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.711 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.712 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.712 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.712 INFO:teuthology.orchestra.run.smithi012.stdout: "daemon_cache_timeout": { 2024-03-18T00:06:44.712 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "daemon_cache_timeout", 2024-03-18T00:06:44.712 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.712 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.712 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.712 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:44.712 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.712 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.712 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.712 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T00:06:44.712 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.713 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.713 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.713 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.713 INFO:teuthology.orchestra.run.smithi012.stdout: "default_cephadm_command_timeout": { 2024-03-18T00:06:44.713 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T00:06:44.713 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.713 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.713 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.713 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "900", 2024-03-18T00:06:44.713 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.713 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.713 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.713 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T00:06:44.714 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.714 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.714 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.714 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.714 INFO:teuthology.orchestra.run.smithi012.stdout: "default_registry": { 2024-03-18T00:06:44.714 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_registry", 2024-03-18T00:06:44.714 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.714 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.714 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.714 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io", 2024-03-18T00:06:44.714 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.714 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.714 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: "device_cache_timeout": { 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_cache_timeout", 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1800", 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.715 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.716 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache device inventory", 2024-03-18T00:06:44.716 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.716 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.716 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.716 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.716 INFO:teuthology.orchestra.run.smithi012.stdout: "device_enhanced_scan": { 2024-03-18T00:06:44.716 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_enhanced_scan", 2024-03-18T00:06:44.716 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.716 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.716 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.716 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.716 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.717 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.717 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.717 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T00:06:44.717 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.717 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.717 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.717 INFO:teuthology.orchestra.run.smithi012.stdout: "facts_cache_timeout": { 2024-03-18T00:06:44.718 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "facts_cache_timeout", 2024-03-18T00:06:44.718 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.718 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.718 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.718 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:44.718 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.718 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.719 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.719 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache host facts data", 2024-03-18T00:06:44.719 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.719 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.719 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.719 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.719 INFO:teuthology.orchestra.run.smithi012.stdout: "host_check_interval": { 2024-03-18T00:06:44.719 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "host_check_interval", 2024-03-18T00:06:44.720 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.720 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.720 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.720 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:44.720 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.720 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.720 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.721 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to perform a host check", 2024-03-18T00:06:44.721 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.721 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.721 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.721 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.721 INFO:teuthology.orchestra.run.smithi012.stdout: "hw_monitoring": { 2024-03-18T00:06:44.722 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hw_monitoring", 2024-03-18T00:06:44.722 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.722 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.722 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.722 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.722 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.723 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.723 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.723 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T00:06:44.723 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.723 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.723 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.723 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.723 INFO:teuthology.orchestra.run.smithi012.stdout: "inventory_list_all": { 2024-03-18T00:06:44.723 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "inventory_list_all", 2024-03-18T00:06:44.723 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.724 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.724 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.724 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.724 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.724 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.724 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.725 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T00:06:44.725 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.725 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.725 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.725 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.725 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.725 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.725 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.725 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.726 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.726 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.726 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.726 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.726 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.726 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.726 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.726 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.727 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.727 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.727 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.727 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.727 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.727 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.727 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.727 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.727 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: "log_refresh_metadata": { 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_refresh_metadata", 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.728 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.729 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.730 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.731 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.732 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.733 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.733 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.733 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.733 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T00:06:44.733 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.733 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.733 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.733 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T00:06:44.733 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T00:06:44.733 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.733 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.733 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.734 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "*", 2024-03-18T00:06:44.734 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.734 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.734 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.734 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T00:06:44.734 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.734 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.734 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.734 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.734 INFO:teuthology.orchestra.run.smithi012.stdout: "max_count_per_host": { 2024-03-18T00:06:44.734 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_count_per_host", 2024-03-18T00:06:44.734 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.734 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of daemons per service per host", 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: "max_osd_draining_count": { 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_osd_draining_count", 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.735 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "migration_current": { 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "migration_current", 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.736 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "internal - do not modify", 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.737 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.738 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "root", 2024-03-18T00:06:44.738 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.738 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.738 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.738 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm-package", 2024-03-18T00:06:44.738 INFO:teuthology.orchestra.run.smithi012.stdout: "root" 2024-03-18T00:06:44.738 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.738 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T00:06:44.738 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.738 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.738 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.738 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.738 INFO:teuthology.orchestra.run.smithi012.stdout: "oob_default_addr": { 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "oob_default_addr", 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "169.254.1.1", 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.739 INFO:teuthology.orchestra.run.smithi012.stdout: "prometheus_alerts_path": { 2024-03-18T00:06:44.740 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus_alerts_path", 2024-03-18T00:06:44.740 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.740 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.740 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.740 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T00:06:44.740 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.740 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.740 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.740 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T00:06:44.740 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.740 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.740 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.740 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.741 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_insecure": { 2024-03-18T00:06:44.741 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_insecure", 2024-03-18T00:06:44.741 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.741 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.741 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.741 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.741 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.741 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.742 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.742 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T00:06:44.742 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.742 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.742 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.742 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.742 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_password": { 2024-03-18T00:06:44.742 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_password", 2024-03-18T00:06:44.742 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.743 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.743 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.743 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.743 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.743 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.743 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.743 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T00:06:44.743 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.744 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.744 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.744 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_url": { 2024-03-18T00:06:44.744 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_url", 2024-03-18T00:06:44.744 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.744 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.744 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.744 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.744 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_username": { 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_username", 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.745 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: "secure_monitoring_stack": { 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "secure_monitoring_stack", 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: "service_discovery_port": { 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "service_discovery_port", 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8765", 2024-03-18T00:06:44.747 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "cephadm service discovery port", 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: "ssh_config_file": { 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssh_config_file", 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.748 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.749 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.749 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.749 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T00:06:44.749 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.749 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.749 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.749 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.749 INFO:teuthology.orchestra.run.smithi012.stdout: "use_agent": { 2024-03-18T00:06:44.749 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_agent", 2024-03-18T00:06:44.749 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.749 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.749 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.749 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.750 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.750 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.750 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.750 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T00:06:44.750 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.750 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.750 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.750 INFO:teuthology.orchestra.run.smithi012.stdout: "use_repo_digest": { 2024-03-18T00:06:44.750 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_repo_digest", 2024-03-18T00:06:44.750 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.750 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.750 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_failed_host_check": { 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_failed_host_check", 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.751 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_daemons": { 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_daemons", 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.752 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.753 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.753 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.753 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.753 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.753 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T00:06:44.753 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.753 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.753 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.753 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_hosts": { 2024-03-18T00:06:44.753 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_hosts", 2024-03-18T00:06:44.753 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.753 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.754 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crash", 2024-03-18T00:06:44.755 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.755 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.755 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.755 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.755 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.755 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.755 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.755 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.755 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.755 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.755 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.755 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.755 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.756 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.757 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.758 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.758 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.758 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.758 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.758 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.758 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.758 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.759 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.759 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.759 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.759 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.759 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.759 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.759 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.759 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.759 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.760 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.760 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.760 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.760 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.760 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.760 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.760 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.760 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.760 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.761 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.761 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.761 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.761 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.761 INFO:teuthology.orchestra.run.smithi012.stdout: "retain_interval": { 2024-03-18T00:06:44.761 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retain_interval", 2024-03-18T00:06:44.761 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.761 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.761 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.761 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "31536000", 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_recent_interval": { 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_recent_interval", 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.762 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1209600", 2024-03-18T00:06:44.763 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.763 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.763 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.763 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T00:06:44.763 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.763 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.763 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.763 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.763 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.763 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.763 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.763 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dashboard", 2024-03-18T00:06:44.763 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.764 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.765 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.766 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.767 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.767 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.767 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_ENABLED": { 2024-03-18T00:06:44.767 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T00:06:44.767 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.767 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.767 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.767 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.767 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.767 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.767 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.767 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.768 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.769 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.770 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.770 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.770 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.770 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.770 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T00:06:44.770 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T00:06:44.770 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.770 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.770 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.770 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.770 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.770 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.770 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.771 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.772 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.772 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.772 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.772 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.772 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.772 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T00:06:44.772 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T00:06:44.772 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.772 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.772 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.772 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.772 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.772 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.773 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.774 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.774 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.774 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.774 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.774 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.774 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.774 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T00:06:44.774 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T00:06:44.774 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.774 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.774 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.775 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.776 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.776 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.776 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.776 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.776 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.776 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.776 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.777 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.777 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.777 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T00:06:44.777 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T00:06:44.777 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.777 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.777 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.777 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.778 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.778 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.778 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.778 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.778 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.778 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.778 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.778 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.779 INFO:teuthology.orchestra.run.smithi012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T00:06:44.779 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T00:06:44.779 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.779 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.779 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.779 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.779 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.779 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.779 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.780 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_URL": { 2024-03-18T00:06:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_URL", 2024-03-18T00:06:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T00:06:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T00:06:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T00:06:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T00:06:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "{}", 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T00:06:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T00:06:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T00:06:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T00:06:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T00:06:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T00:06:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T00:06:44.794 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T00:06:44.794 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.794 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.794 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.794 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.794 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.794 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.794 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.795 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.795 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.795 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.795 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.795 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T00:06:44.795 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T00:06:44.795 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.795 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.796 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.796 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.796 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.796 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.796 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.796 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.796 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.796 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.797 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.798 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.799 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.800 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8", 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T00:06:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T00:06:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "45", 2024-03-18T00:06:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.804 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.805 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.806 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.806 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T00:06:44.806 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T00:06:44.806 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.806 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.806 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.806 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-03-18T00:06:44.806 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.806 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.806 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.806 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.806 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.806 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.807 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.808 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.809 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.809 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.809 INFO:teuthology.orchestra.run.smithi012.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T00:06:44.809 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T00:06:44.809 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.809 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.809 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.809 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.809 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.809 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.809 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.809 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.809 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.810 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.810 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.810 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T00:06:44.810 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T00:06:44.810 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.810 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.810 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.810 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:44.810 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.810 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.810 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.811 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.812 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.812 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.812 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.812 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.812 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.812 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T00:06:44.812 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T00:06:44.813 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.813 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.813 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.813 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-18T00:06:44.813 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.813 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.813 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.813 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.814 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.814 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.814 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.814 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.814 INFO:teuthology.orchestra.run.smithi012.stdout: "cross_origin_url": { 2024-03-18T00:06:44.814 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cross_origin_url", 2024-03-18T00:06:44.814 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.814 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.815 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.815 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.815 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.815 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.815 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.815 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.815 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.815 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.815 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "crt_file": { 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crt_file", 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: "debug": { 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "debug", 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable/disable debug options", 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.817 INFO:teuthology.orchestra.run.smithi012.stdout: "jwt_token_ttl": { 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "jwt_token_ttl", 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "28800", 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.818 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-03-18T00:06:44.819 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-03-18T00:06:44.819 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.819 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.819 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.819 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.819 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.819 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.819 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.819 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.819 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.819 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.819 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.819 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.820 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.821 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.821 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.821 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.821 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.821 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.821 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.821 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.821 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.821 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.821 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.821 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.821 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.821 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.822 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.822 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.822 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.822 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.822 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.822 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.822 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.822 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.822 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.822 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.822 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.822 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.822 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.823 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.823 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.823 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.823 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.823 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.823 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.823 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.823 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.823 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.823 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.823 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.823 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: "motd": { 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "motd", 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.825 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "The message of the day", 2024-03-18T00:06:44.826 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.826 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.826 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.826 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_resolve_ip_addr": { 2024-03-18T00:06:44.826 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T00:06:44.826 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.826 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.826 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.826 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.826 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.826 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.826 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.827 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.828 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.828 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.828 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.828 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.828 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-18T00:06:44.828 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-18T00:06:44.828 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.828 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.828 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.828 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8080", 2024-03-18T00:06:44.828 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.828 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.829 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.830 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.830 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.830 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.830 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.830 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.830 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl_server_port": { 2024-03-18T00:06:44.830 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl_server_port", 2024-03-18T00:06:44.830 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.830 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8443", 2024-03-18T00:06:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-03-18T00:06:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-03-18T00:06:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "redirect", 2024-03-18T00:06:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect" 2024-03-18T00:06:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-03-18T00:06:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-03-18T00:06:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-03-18T00:06:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-03-18T00:06:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-03-18T00:06:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "url_prefix": { 2024-03-18T00:06:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url_prefix", 2024-03-18T00:06:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_monitoring": { 2024-03-18T00:06:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_monitoring", 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "monitor device health metrics", 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "mark_out_threshold": { 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mark_out_threshold", 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2419200", 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_name": { 2024-03-18T00:06:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_name", 2024-03-18T00:06:44.844 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.844 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.844 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.844 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "device_health_metrics", 2024-03-18T00:06:44.844 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.844 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.844 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.844 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T00:06:44.844 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.844 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.844 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.844 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.844 INFO:teuthology.orchestra.run.smithi012.stdout: "retention_period": { 2024-03-18T00:06:44.845 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retention_period", 2024-03-18T00:06:44.845 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.845 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.845 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.845 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15552000", 2024-03-18T00:06:44.845 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.845 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.845 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.845 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain device health metrics", 2024-03-18T00:06:44.845 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.845 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.845 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_frequency": { 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_frequency", 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.847 INFO:teuthology.orchestra.run.smithi012.stdout: "self_heal": { 2024-03-18T00:06:44.847 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "self_heal", 2024-03-18T00:06:44.847 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.847 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.847 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.847 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.847 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.847 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.847 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.847 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T00:06:44.847 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.847 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.847 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.848 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.848 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:44.848 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:44.848 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.848 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.848 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.848 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:44.848 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.848 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.848 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.848 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T00:06:44.849 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.849 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.849 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.849 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.849 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_threshold": { 2024-03-18T00:06:44.849 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_threshold", 2024-03-18T00:06:44.849 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.849 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.849 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.850 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7257600", 2024-03-18T00:06:44.850 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.850 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.850 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.850 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T00:06:44.850 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.850 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.850 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.851 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.851 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.851 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.851 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.851 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "diskprediction_local", 2024-03-18T00:06:44.851 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.851 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.851 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.852 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.853 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.854 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.855 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.855 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.855 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.855 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.855 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.855 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.855 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.855 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.855 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.855 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.855 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.855 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.855 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.856 INFO:teuthology.orchestra.run.smithi012.stdout: "predict_interval": { 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predict_interval", 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.857 INFO:teuthology.orchestra.run.smithi012.stdout: "predictor_model": { 2024-03-18T00:06:44.858 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predictor_model", 2024-03-18T00:06:44.858 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.858 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.858 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.858 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "prophetstor", 2024-03-18T00:06:44.858 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.858 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.858 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.858 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.858 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.858 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.859 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.859 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:44.859 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:44.859 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.859 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.859 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.859 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-03-18T00:06:44.859 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.859 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.859 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.859 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.859 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "influx", 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": false, 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "influxdb python module not found", 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: "batch_size": { 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "batch_size", 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.860 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5000", 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "database": { 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "database", 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.861 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "ceph", 2024-03-18T00:06:44.862 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.862 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.862 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.862 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T00:06:44.862 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.862 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.862 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.862 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": { 2024-03-18T00:06:44.862 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hostname", 2024-03-18T00:06:44.862 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server hostname", 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:44.863 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "30", 2024-03-18T00:06:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "5", 2024-03-18T00:06:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T00:06:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.865 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.865 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.865 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.865 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.865 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.865 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.865 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.865 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.865 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.865 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.865 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.865 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.865 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.866 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.867 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.867 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.867 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.867 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.867 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.867 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.867 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.867 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.867 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.867 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.867 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.869 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.869 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.869 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.869 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.869 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.869 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.870 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.870 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.870 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.870 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.870 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.870 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.870 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.870 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.871 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.871 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.871 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.871 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.871 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.871 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.871 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.871 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.872 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.872 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.872 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.872 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.872 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.872 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.872 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.872 INFO:teuthology.orchestra.run.smithi012.stdout: "password": { 2024-03-18T00:06:44.872 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "password", 2024-03-18T00:06:44.872 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.872 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.872 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "password of InfluxDB server user", 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: "port": { 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "port", 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.873 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8086", 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server port", 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.874 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "false", 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: "threads": { 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threads", 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.875 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "32", 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "username": { 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "username", 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.876 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.877 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.877 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.877 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "username of InfluxDB server user", 2024-03-18T00:06:44.877 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.877 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.877 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.877 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.877 INFO:teuthology.orchestra.run.smithi012.stdout: "verify_ssl": { 2024-03-18T00:06:44.877 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "verify_ssl", 2024-03-18T00:06:44.877 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.877 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.877 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.877 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "true", 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "insights", 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.878 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.879 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.880 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.881 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.882 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.883 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "iostat", 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.884 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.885 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.887 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.887 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.887 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.887 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.887 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.887 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.887 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.887 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.887 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.887 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.887 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.887 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.887 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.888 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.889 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.889 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.889 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.889 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.889 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.889 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.889 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.889 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.889 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.890 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.890 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.890 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.890 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "k8sevents", 2024-03-18T00:06:44.890 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.890 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.890 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.890 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_event_retention_days": { 2024-03-18T00:06:44.891 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ceph_event_retention_days", 2024-03-18T00:06:44.891 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.891 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.891 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.891 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7", 2024-03-18T00:06:44.891 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.891 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T00:06:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: "config_check_secs": { 2024-03-18T00:06:44.892 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_check_secs", 2024-03-18T00:06:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "10", 2024-03-18T00:06:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T00:06:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.893 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.894 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.895 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.896 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.896 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.896 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.896 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.896 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.896 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.896 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.896 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.896 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.896 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.896 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.896 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.896 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.897 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.897 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.897 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.897 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.897 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.897 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.897 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.897 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.897 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.897 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.897 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.897 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.897 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.898 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.898 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.898 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.898 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.898 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.898 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.898 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.898 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.898 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.898 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.898 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.899 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.899 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.899 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.899 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.899 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.899 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.899 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.899 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "localpool", 2024-03-18T00:06:44.899 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.899 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.899 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.899 INFO:teuthology.orchestra.run.smithi012.stdout: "failure_domain": { 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "failure_domain", 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "host", 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "failure domain for any created local pool", 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.900 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.901 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.901 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.901 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.901 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.901 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.901 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.901 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.901 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.901 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.901 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.901 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.901 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.902 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.902 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.902 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.902 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.902 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.902 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.902 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.902 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.902 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.902 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.902 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.902 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.903 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.903 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.903 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.903 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.903 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.903 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.903 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.903 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.903 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.903 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.903 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.904 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.904 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.904 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.904 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.904 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.904 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.904 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.904 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.904 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.904 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.904 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.905 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.906 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.906 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.906 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.906 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.906 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.906 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.906 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.906 INFO:teuthology.orchestra.run.smithi012.stdout: "min_size": { 2024-03-18T00:06:44.906 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_size", 2024-03-18T00:06:44.906 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.906 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.906 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default min_size for any created local pool", 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: "num_rep": { 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "num_rep", 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.907 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.908 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3", 2024-03-18T00:06:44.908 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.908 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.908 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.908 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default replica count for any created local pool", 2024-03-18T00:06:44.908 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.908 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.909 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.909 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_num": { 2024-03-18T00:06:44.909 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_num", 2024-03-18T00:06:44.909 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.909 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.909 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.909 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "128", 2024-03-18T00:06:44.909 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.910 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.910 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.910 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T00:06:44.910 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.910 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.910 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.910 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.910 INFO:teuthology.orchestra.run.smithi012.stdout: "prefix": { 2024-03-18T00:06:44.911 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prefix", 2024-03-18T00:06:44.911 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.911 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.911 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.911 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.911 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.911 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.911 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.911 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name prefix for any created local pool", 2024-03-18T00:06:44.911 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.911 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.911 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.911 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: "subtree": { 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "subtree", 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "rack", 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.912 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mds_autoscaler", 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.913 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.914 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.915 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.916 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.917 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirroring", 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.918 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.919 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.920 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "nfs", 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.926 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.927 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.928 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.929 INFO:teuthology.orchestra.run.smithi012.stdout: "fail_fs": { 2024-03-18T00:06:44.930 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "fail_fs", 2024-03-18T00:06:44.930 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.930 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.930 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.930 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.930 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.930 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.930 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.930 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T00:06:44.930 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.930 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.930 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.930 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator": { 2024-03-18T00:06:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-03-18T00:06:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-03-18T00:06:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "rook", 2024-03-18T00:06:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "test_orchestrator" 2024-03-18T00:06:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Orchestrator backend", 2024-03-18T00:06:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_perf_query", 2024-03-18T00:06:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_support", 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.944 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.944 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.944 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.944 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.944 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.944 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.944 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.944 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.944 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.944 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.945 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.945 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.945 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.945 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.945 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.945 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.945 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.945 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.945 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.946 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.946 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.946 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.946 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.946 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.946 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.946 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.946 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.946 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.946 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.946 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.946 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.947 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.948 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.949 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_autoscaler", 2024-03-18T00:06:44.950 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.950 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.950 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.950 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.950 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.950 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.950 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.950 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.950 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.950 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.950 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.950 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.951 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.952 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.952 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.952 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.952 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.952 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.952 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.952 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.952 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.952 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.952 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.952 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.952 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.953 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.954 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.954 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.954 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.954 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.954 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.954 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.954 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.954 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.954 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.954 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.954 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.954 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.954 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:44.955 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: "threshold": { 2024-03-18T00:06:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threshold", 2024-03-18T00:06:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.956 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-03-18T00:06:44.957 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1.0", 2024-03-18T00:06:44.957 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.957 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.957 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "scaling threshold", 2024-03-18T00:06:44.957 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T00:06:44.957 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.957 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.957 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.957 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.957 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.957 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "progress", 2024-03-18T00:06:44.957 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.958 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.958 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.958 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_pg_recovery_event": { 2024-03-18T00:06:44.958 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_pg_recovery_event", 2024-03-18T00:06:44.958 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.958 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.958 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.958 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.958 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.958 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.958 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.958 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.959 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.960 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.961 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.961 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.961 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.961 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.961 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.961 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.961 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.961 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.961 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.961 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.961 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.961 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.961 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.962 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.963 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.964 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.964 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.964 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.964 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.964 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.964 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.964 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.964 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.964 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.964 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.964 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.964 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.964 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.965 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.965 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.965 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.965 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.965 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.965 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.965 INFO:teuthology.orchestra.run.smithi012.stdout: "max_completed_events": { 2024-03-18T00:06:44.965 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_completed_events", 2024-03-18T00:06:44.965 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.965 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.965 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.965 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "50", 2024-03-18T00:06:44.966 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.966 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.966 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.966 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "number of past completed events to remember", 2024-03-18T00:06:44.966 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.966 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.966 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.966 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.966 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-03-18T00:06:44.966 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-03-18T00:06:44.966 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:44.966 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.966 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long the module is going to sleep", 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus", 2024-03-18T00:06:44.967 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "cache": { 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cache", 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.968 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.969 INFO:teuthology.orchestra.run.smithi012.stdout: "exclude_perf_counters": { 2024-03-18T00:06:44.969 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "exclude_perf_counters", 2024-03-18T00:06:44.969 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.969 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.969 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.969 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.969 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.969 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.969 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.969 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T00:06:44.969 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T00:06:44.969 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.969 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.970 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.971 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.971 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.971 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.971 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.971 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.971 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.971 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.971 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.971 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.971 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.971 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.972 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.973 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.973 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.973 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.973 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.973 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.973 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.973 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.973 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.973 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.973 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.973 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.973 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.974 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.974 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.974 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.974 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.974 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.974 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.974 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.974 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.974 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.974 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.974 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools": { 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools", 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.975 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.976 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "300", 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_interval": { 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_interval", 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15.0", 2024-03-18T00:06:44.977 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.978 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.978 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.978 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.978 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.978 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.978 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.978 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.978 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-18T00:06:44.978 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-18T00:06:44.978 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.978 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.978 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.978 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "9283", 2024-03-18T00:06:44.979 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: "stale_cache_strategy": { 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stale_cache_strategy", 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "log", 2024-03-18T00:06:44.980 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "default", 2024-03-18T00:06:44.981 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: "default", 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: "error" 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.982 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-03-18T00:06:44.983 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.984 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.985 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.986 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.987 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.988 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_snap_create": { 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_snap_create", 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.989 INFO:teuthology.orchestra.run.smithi012.stdout: "mirror_snapshot_schedule": { 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.990 INFO:teuthology.orchestra.run.smithi012.stdout: "trash_purge_schedule": { 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "trash_purge_schedule", 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:44.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "restful", 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_auth": { 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_auth", 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.992 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.993 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.994 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:44.995 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:44.996 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:44.997 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.998 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:44.999 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.000 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rgw", 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.001 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.002 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.002 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.002 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.002 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.002 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.002 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.002 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.002 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.002 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.002 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.002 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.002 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.002 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.003 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.004 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:45.005 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.006 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.007 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.007 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rook", 2024-03-18T00:06:45.007 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:45.007 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:45.007 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:45.007 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:45.007 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:45.007 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.007 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.007 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.007 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.007 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.007 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.008 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.008 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.008 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.008 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.008 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.008 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.008 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.008 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.008 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.008 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.008 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.008 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.008 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_class": { 2024-03-18T00:06:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "storage_class", 2024-03-18T00:06:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "local", 2024-03-18T00:06:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T00:06:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "selftest", 2024-03-18T00:06:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "roption1": { 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption1", 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "roption2": { 2024-03-18T00:06:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption2", 2024-03-18T00:06:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "xyz", 2024-03-18T00:06:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption1": { 2024-03-18T00:06:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption1", 2024-03-18T00:06:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption2": { 2024-03-18T00:06:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption2", 2024-03-18T00:06:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption3": { 2024-03-18T00:06:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption3", 2024-03-18T00:06:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-03-18T00:06:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption4": { 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption4", 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.024 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption5": { 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption5", 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.025 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption6": { 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption6", 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.026 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption7": { 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption7", 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.027 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.028 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-03-18T00:06:45.028 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "42", 2024-03-18T00:06:45.028 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.028 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.028 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.028 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.028 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.028 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.028 INFO:teuthology.orchestra.run.smithi012.stdout: "testkey": { 2024-03-18T00:06:45.028 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testkey", 2024-03-18T00:06:45.028 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.028 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.028 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.029 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.029 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.029 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.029 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.029 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.029 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.029 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.029 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.029 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.029 INFO:teuthology.orchestra.run.smithi012.stdout: "testlkey": { 2024-03-18T00:06:45.030 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testlkey", 2024-03-18T00:06:45.030 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.030 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.030 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.030 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.030 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.030 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.030 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.030 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.030 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.030 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.030 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.030 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.031 INFO:teuthology.orchestra.run.smithi012.stdout: "testnewline": { 2024-03-18T00:06:45.031 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testnewline", 2024-03-18T00:06:45.031 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.031 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.031 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.031 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.031 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.031 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.031 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.031 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_m_granularity": { 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_m_granularity", 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.032 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: "dump_on_update": { 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dump_on_update", 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.033 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "dump database to debug log on update", 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.034 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.035 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.036 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.037 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.038 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.038 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.038 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.038 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.038 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.038 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.038 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.038 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.038 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:45.038 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:45.038 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.038 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.038 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stats", 2024-03-18T00:06:45.039 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.040 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.041 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.042 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.042 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.042 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.042 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.042 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.042 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.042 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.042 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:45.042 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:45.042 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.042 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.042 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.043 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.044 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "status", 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.045 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.046 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.047 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.047 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:45.047 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:45.047 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.047 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.047 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.047 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.047 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.047 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.047 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.047 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.047 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.047 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.048 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.049 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telegraf", 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:45.050 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "address": { 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "address", 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.051 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15", 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.052 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.053 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.054 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.055 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.056 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.057 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telemetry", 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_basic": { 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_basic", 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.058 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_crash": { 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_crash", 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.059 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_device": { 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_device", 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.060 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_ident": { 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_ident", 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.061 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_perf": { 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_perf", 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.062 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: "contact": { 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "contact", 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.063 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: "description": { 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "description", 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.064 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: "device_url": { 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_url", 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.065 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T00:06:45.066 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.066 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.066 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.066 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.066 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.066 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.066 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.066 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-03-18T00:06:45.066 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-03-18T00:06:45.066 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.066 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.066 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.067 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "24", 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "8", 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: "last_opt_revision": { 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "last_opt_revision", 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.068 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.069 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1", 2024-03-18T00:06:45.069 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.069 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.069 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.069 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.069 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.069 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.069 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.069 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard": { 2024-03-18T00:06:45.069 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard", 2024-03-18T00:06:45.069 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.069 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard_description": { 2024-03-18T00:06:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard_description", 2024-03-18T00:06:45.070 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:45.071 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.072 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.072 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.072 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.072 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.072 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.072 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.072 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.072 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.072 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.072 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.072 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.072 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.072 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.073 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.074 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.074 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.074 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.074 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.074 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.074 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:45.074 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:45.074 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.074 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.074 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.074 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:45.074 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.074 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.075 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:45.076 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:45.076 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.076 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.076 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.076 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.076 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "organization": { 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "organization", 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.077 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: "proxy": { 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "proxy", 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "url": { 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url", 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.079 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test_orchestrator", 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:45.080 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.081 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.082 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.083 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.084 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-03-18T00:06:45.085 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.090 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_clones": { 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_clones", 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4", 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.091 INFO:teuthology.orchestra.run.smithi012.stdout: "periodic_async_work": { 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "periodic_async_work", 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Periodically check for async work", 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.092 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_delay": { 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_delay", 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_no_wait": { 2024-03-18T00:06:45.093 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.094 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix", 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "discovery_interval": { 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "discovery_interval", 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "100", 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.095 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: "identifier": { 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "identifier", 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.096 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.097 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.098 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.099 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.100 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.101 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.102 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_host": { 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_host", 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.103 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_port": { 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_port", 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10051", 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.104 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_sender": { 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_sender", 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-03-18T00:06:45.105 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: "services": { 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: "dashboard": "https://172.21.15.12:8443/" 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: "always_on_modules": { 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus": [ 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-03-18T00:06:45.106 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific": [ 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-03-18T00:06:45.107 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy": [ 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-03-18T00:06:45.108 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.109 INFO:teuthology.orchestra.run.smithi012.stdout: "reef": [ 2024-03-18T00:06:45.109 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-03-18T00:06:45.109 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-03-18T00:06:45.109 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-03-18T00:06:45.109 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-03-18T00:06:45.109 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-03-18T00:06:45.109 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-03-18T00:06:45.109 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-03-18T00:06:45.109 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-03-18T00:06:45.109 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-03-18T00:06:45.109 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-03-18T00:06:45.109 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-03-18T00:06:45.109 INFO:teuthology.orchestra.run.smithi012.stdout: "squid": [ 2024-03-18T00:06:45.110 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-03-18T00:06:45.110 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-03-18T00:06:45.110 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-03-18T00:06:45.110 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-03-18T00:06:45.110 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-03-18T00:06:45.110 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-03-18T00:06:45.110 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-03-18T00:06:45.110 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-03-18T00:06:45.110 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-03-18T00:06:45.110 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-03-18T00:06:45.110 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-18T00:06:45.110 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.110 INFO:teuthology.orchestra.run.smithi012.stdout: "last_failure_osd_epoch": 3, 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: "active_clients": [ 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1070514851 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "libcephsqlite", 2024-03-18T00:06:45.111 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1849534255 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-03-18T00:06:45.112 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3450532080 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1081321261 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:45.113 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-18T00:06:45.114 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-18T00:06:45.115 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:43 smithi079 ceph-mon[36241]: pgmap v678: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 452 op/s 2024-03-18T00:06:45.115 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:43 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3983744729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:06:45.115 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:43 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1953950439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T00:06:45.116 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:44 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1018618744' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T00:06:45.116 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:43 smithi191 ceph-mon[36708]: pgmap v678: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 452 op/s 2024-03-18T00:06:45.116 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:43 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3983744729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:06:45.116 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:43 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1953950439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T00:06:45.116 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:44 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1018618744' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T00:06:45.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.769+0000 7fea6b600640 1 -- 172.21.15.12:0/1018618744 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fea480a2800 msgr2=0x7fea480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:45.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.769+0000 7fea6b600640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fea480a2800 0x7fea480a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fea64193350 tx=0x7fea4c005f90 comp rx=0 tx=0).stop 2024-03-18T00:06:45.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.769+0000 7fea6b600640 1 -- 172.21.15.12:0/1018618744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea64100700 msgr2=0x7fea641953d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:06:45.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.769+0000 7fea6b600640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea64100700 0x7fea641953d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fea5400e9c0 tx=0x7fea5400ee80 comp rx=0 tx=0).stop 2024-03-18T00:06:45.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.769+0000 7fea6b600640 1 -- 172.21.15.12:0/1018618744 shutdown_connections 2024-03-18T00:06:45.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.769+0000 7fea6b600640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fea480a2800 0x7fea480a4cc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:45.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.769+0000 7fea6b600640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea641022b0 0x7fea64191f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:45.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.769+0000 7fea6b600640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea641018f0 0x7fea64195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:45.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.769+0000 7fea6b600640 1 --2- 172.21.15.12:0/1018618744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea64100700 0x7fea641953d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:06:45.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.769+0000 7fea6b600640 1 -- 172.21.15.12:0/1018618744 >> 172.21.15.12:0/1018618744 conn(0x7fea640fbc70 msgr2=0x7fea640fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:06:45.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.770+0000 7fea6b600640 1 -- 172.21.15.12:0/1018618744 shutdown_connections 2024-03-18T00:06:45.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:06:43.770+0000 7fea6b600640 1 -- 172.21.15.12:0/1018618744 wait complete. 2024-03-18T00:06:45.123 DEBUG:tasks.check_counter:Getting stats from a 2024-03-18T00:06:45.124 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-18T00:06:45.124 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:43 smithi012 ceph-mon[31541]: pgmap v678: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 452 op/s 2024-03-18T00:06:45.124 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:43 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3983744729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:06:45.124 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:43 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1953950439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T00:06:45.125 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:44 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1018618744' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T00:06:45.586 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:06:45.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:45 smithi191 ceph-mon[36708]: pgmap v679: 129 pgs: 129 active+clean; 232 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 877 KiB/s wr, 434 op/s 2024-03-18T00:06:45.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:45 smithi012 ceph-mon[31541]: pgmap v679: 129 pgs: 129 active+clean; 232 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 877 KiB/s wr, 434 op/s 2024-03-18T00:06:46.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:45 smithi079 ceph-mon[36241]: pgmap v679: 129 pgs: 129 active+clean; 232 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 877 KiB/s wr, 434 op/s 2024-03-18T00:06:47.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:47 smithi191 ceph-mon[36708]: pgmap v680: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 885 KiB/s wr, 369 op/s 2024-03-18T00:06:47.885 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:47 smithi012 ceph-mon[31541]: pgmap v680: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 885 KiB/s wr, 369 op/s 2024-03-18T00:06:48.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:47 smithi079 ceph-mon[36241]: pgmap v680: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 885 KiB/s wr, 369 op/s 2024-03-18T00:06:48.260 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-18T00:06:48.260 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T00:06:48.260 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 2842, 2024-03-18T00:06:48.260 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 2776, 2024-03-18T00:06:48.260 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 1762833, 2024-03-18T00:06:48.260 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 10031089, 2024-03-18T00:06:48.260 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 9, 2024-03-18T00:06:48.260 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 5, 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.565700222, 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.198721415, 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.243079281, 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.121755297, 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 2775, 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.945298792, 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000340648 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 2842, 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000919516, 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000323 2024-03-18T00:06:48.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.262 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 920304, 2024-03-18T00:06:48.262 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 126272 2024-03-18T00:06:48.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.262 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T00:06:48.262 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 17026, 2024-03-18T00:06:48.262 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 22367, 2024-03-18T00:06:48.262 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 5980433, 2024-03-18T00:06:48.262 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 11003773, 2024-03-18T00:06:48.262 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 15, 2024-03-18T00:06:48.262 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 9, 2024-03-18T00:06:48.262 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 6.835021038, 2024-03-18T00:06:48.262 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 5.080994500, 2024-03-18T00:06:48.262 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 2.252745135, 2024-03-18T00:06:48.263 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.055596102, 2024-03-18T00:06:48.263 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:06:48.263 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 22365, 2024-03-18T00:06:48.263 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 10.247970031, 2024-03-18T00:06:48.263 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000458214 2024-03-18T00:06:48.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.263 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:06:48.263 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 31705, 2024-03-18T00:06:48.263 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.027747640, 2024-03-18T00:06:48.263 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000875 2024-03-18T00:06:48.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.263 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-03-18T00:06:48.263 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 335216 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 17102, 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 15689, 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 7319205, 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 4467876, 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 8, 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 6, 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 2.818154999, 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.993098148, 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 1.860748525, 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.102496829, 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:06:48.264 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 15688, 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 4.974222053, 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000317071 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 29041, 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.014448377, 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000497 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: "cct": { 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: "total_workers": 1, 2024-03-18T00:06:48.265 INFO:teuthology.orchestra.run.smithi012.stdout: "unhealthy_workers": 0 2024-03-18T00:06:48.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.266 INFO:teuthology.orchestra.run.smithi012.stdout: "finisher-MDSRank": { 2024-03-18T00:06:48.266 INFO:teuthology.orchestra.run.smithi012.stdout: "queue_len": 0, 2024-03-18T00:06:48.266 INFO:teuthology.orchestra.run.smithi012.stdout: "complete_latency": { 2024-03-18T00:06:48.266 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 911, 2024-03-18T00:06:48.266 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 8.973385297, 2024-03-18T00:06:48.266 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.009850038 2024-03-18T00:06:48.266 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:48.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.266 INFO:teuthology.orchestra.run.smithi012.stdout: "finisher-PurgeQueue": { 2024-03-18T00:06:48.266 INFO:teuthology.orchestra.run.smithi012.stdout: "queue_len": 0, 2024-03-18T00:06:48.266 INFO:teuthology.orchestra.run.smithi012.stdout: "complete_latency": { 2024-03-18T00:06:48.266 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 2250, 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.649352380, 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000288601 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": { 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: "request": 8442, 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: "reply": 7873, 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: "reply_latency": { 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 7873, 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 134.556715754, 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.017090907 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: "slow_reply": 0, 2024-03-18T00:06:48.267 INFO:teuthology.orchestra.run.smithi012.stdout: "forward": 569, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_fetch_complete": 0, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_fetch_keys": 0, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_commit": 47, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_split": 4, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_merge": 0, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes": 27, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_top": 8, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_bottom": 9, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_pin_tail": 10, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_pinned": 13, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_expired": 496, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_with_caps": 2, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": 2, 2024-03-18T00:06:48.268 INFO:teuthology.orchestra.run.smithi012.stdout: "subtrees": 4, 2024-03-18T00:06:48.269 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse": 20136, 2024-03-18T00:06:48.269 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_hit": 16093, 2024-03-18T00:06:48.269 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_forward": 569, 2024-03-18T00:06:48.269 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_discover": 240, 2024-03-18T00:06:48.269 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_dir_fetch": 0, 2024-03-18T00:06:48.269 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_remote_ino": 0, 2024-03-18T00:06:48.269 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_lock": 285, 2024-03-18T00:06:48.269 INFO:teuthology.orchestra.run.smithi012.stdout: "load_cent": 0, 2024-03-18T00:06:48.269 INFO:teuthology.orchestra.run.smithi012.stdout: "q": 0, 2024-03-18T00:06:48.269 INFO:teuthology.orchestra.run.smithi012.stdout: "exported": 32, 2024-03-18T00:06:48.269 INFO:teuthology.orchestra.run.smithi012.stdout: "exported_inodes": 50, 2024-03-18T00:06:48.269 INFO:teuthology.orchestra.run.smithi012.stdout: "imported": 4, 2024-03-18T00:06:48.269 INFO:teuthology.orchestra.run.smithi012.stdout: "imported_inodes": 524, 2024-03-18T00:06:48.270 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_dir_fetch": 0, 2024-03-18T00:06:48.270 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_backtrace_fetch": 0, 2024-03-18T00:06:48.270 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_peer_discover": 0, 2024-03-18T00:06:48.270 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rfiles": 2278, 2024-03-18T00:06:48.270 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rbytes": 28871204, 2024-03-18T00:06:48.270 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rsnaps": 0, 2024-03-18T00:06:48.270 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T00:06:48.270 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_set_tag": 0, 2024-03-18T00:06:48.270 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T00:06:48.270 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_inotable_repaired": 0, 2024-03-18T00:06:48.270 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dir_inodes": 0, 2024-03-18T00:06:48.270 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T00:06:48.270 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_file_inodes": 0, 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_inode_file_caps": 121, 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_revoke": 1659, 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_grant": 2567, 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_caps": 3020, 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_caps_dirty": 1237, 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_cap_release": 162, 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "process_request_cap_release": 4619 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_cache": { 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_update": 629, 2024-03-18T00:06:48.271 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_update_receipt": 4296, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_try_discover": 2700, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_send_discover": 2794, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_handle_discover": 191, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays": 0, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays_delayed": 0, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays_enqueuing": 0, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_created": 1177, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_enqueued": 1177, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_reintegrated": 0, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_migrated": 0, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_processing": 0, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_enqueued": 0, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_prioritized": 0, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "recovery_started": 0, 2024-03-18T00:06:48.272 INFO:teuthology.orchestra.run.smithi012.stdout: "recovery_completed": 0, 2024-03-18T00:06:48.273 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_quiesce_path": 0, 2024-03-18T00:06:48.273 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_quiesce_inode": 0, 2024-03-18T00:06:48.273 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T00:06:48.273 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_exportdir": 32, 2024-03-18T00:06:48.273 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_flush": 0, 2024-03-18T00:06:48.273 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_fragmentdir": 4, 2024-03-18T00:06:48.273 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_fragstats": 0, 2024-03-18T00:06:48.273 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_inodestats": 0 2024-03-18T00:06:48.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.273 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_log": { 2024-03-18T00:06:48.273 INFO:teuthology.orchestra.run.smithi012.stdout: "evlrg": 0, 2024-03-18T00:06:48.273 INFO:teuthology.orchestra.run.smithi012.stdout: "evadd": 6234, 2024-03-18T00:06:48.273 INFO:teuthology.orchestra.run.smithi012.stdout: "evex": 0, 2024-03-18T00:06:48.274 INFO:teuthology.orchestra.run.smithi012.stdout: "evtrm": 0, 2024-03-18T00:06:48.274 INFO:teuthology.orchestra.run.smithi012.stdout: "ev": 6234, 2024-03-18T00:06:48.274 INFO:teuthology.orchestra.run.smithi012.stdout: "evexg": 0, 2024-03-18T00:06:48.274 INFO:teuthology.orchestra.run.smithi012.stdout: "evexd": 0, 2024-03-18T00:06:48.274 INFO:teuthology.orchestra.run.smithi012.stdout: "segadd": 8, 2024-03-18T00:06:48.274 INFO:teuthology.orchestra.run.smithi012.stdout: "segex": 0, 2024-03-18T00:06:48.274 INFO:teuthology.orchestra.run.smithi012.stdout: "segtrm": 0, 2024-03-18T00:06:48.274 INFO:teuthology.orchestra.run.smithi012.stdout: "seg": 8, 2024-03-18T00:06:48.274 INFO:teuthology.orchestra.run.smithi012.stdout: "segmjr": 2, 2024-03-18T00:06:48.274 INFO:teuthology.orchestra.run.smithi012.stdout: "segexg": 0, 2024-03-18T00:06:48.274 INFO:teuthology.orchestra.run.smithi012.stdout: "segexd": 0, 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: "expos": 4194304, 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: "wrpos": 13225601, 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: "rdpos": 0, 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: "jlat": { 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 345, 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 6.898961903, 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.019996991 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: "replayed": 0 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_mem": { 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: "ino": 32, 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: "ino+": 1438, 2024-03-18T00:06:48.275 INFO:teuthology.orchestra.run.smithi012.stdout: "ino-": 1406, 2024-03-18T00:06:48.276 INFO:teuthology.orchestra.run.smithi012.stdout: "dir": 31, 2024-03-18T00:06:48.276 INFO:teuthology.orchestra.run.smithi012.stdout: "dir+": 1585, 2024-03-18T00:06:48.276 INFO:teuthology.orchestra.run.smithi012.stdout: "dir-": 1554, 2024-03-18T00:06:48.276 INFO:teuthology.orchestra.run.smithi012.stdout: "dn": 27, 2024-03-18T00:06:48.276 INFO:teuthology.orchestra.run.smithi012.stdout: "dn+": 2986, 2024-03-18T00:06:48.276 INFO:teuthology.orchestra.run.smithi012.stdout: "dn-": 2959, 2024-03-18T00:06:48.276 INFO:teuthology.orchestra.run.smithi012.stdout: "cap": 2, 2024-03-18T00:06:48.276 INFO:teuthology.orchestra.run.smithi012.stdout: "cap+": 1699, 2024-03-18T00:06:48.276 INFO:teuthology.orchestra.run.smithi012.stdout: "cap-": 1697, 2024-03-18T00:06:48.276 INFO:teuthology.orchestra.run.smithi012.stdout: "rss": 65984, 2024-03-18T00:06:48.276 INFO:teuthology.orchestra.run.smithi012.stdout: "heap": 223612 2024-03-18T00:06:48.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_server": { 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: "dispatch_client_request": 12084, 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: "dispatch_server_request": 50, 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_request": 8442, 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_session": 11, 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_peer_request": 2688, 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: "req_create_latency": { 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 924, 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 12.886380772, 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.013946299 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getattr_latency": { 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1163, 2024-03-18T00:06:48.277 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 19.260365039, 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.016560932 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getfilelock_latency": { 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: "req_link_latency": { 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookup_latency": { 2024-03-18T00:06:48.278 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 3355, 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 85.059663410, 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.025353103 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookuphash_latency": { 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupino_latency": { 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 2, 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000303473, 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000151736 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupname_latency": { 2024-03-18T00:06:48.280 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.280 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.280 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.280 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupparent_latency": { 2024-03-18T00:06:48.280 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.280 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.281 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.281 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupsnap_latency": { 2024-03-18T00:06:48.281 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.281 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.281 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.281 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lssnap_latency": { 2024-03-18T00:06:48.282 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.282 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.282 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.282 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mkdir_latency": { 2024-03-18T00:06:48.282 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.282 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.282 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.282 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mknod_latency": { 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mksnap_latency": { 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: "req_open_latency": { 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 811, 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 2.721946319, 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.003356283 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.283 INFO:teuthology.orchestra.run.smithi012.stdout: "req_readdir_latency": { 2024-03-18T00:06:48.284 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 32, 2024-03-18T00:06:48.284 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.108636376, 2024-03-18T00:06:48.284 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.003394886 2024-03-18T00:06:48.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.284 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rename_latency": { 2024-03-18T00:06:48.284 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 925, 2024-03-18T00:06:48.284 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 12.765691137, 2024-03-18T00:06:48.284 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.013800747 2024-03-18T00:06:48.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.284 INFO:teuthology.orchestra.run.smithi012.stdout: "req_renamesnap_latency": { 2024-03-18T00:06:48.284 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.284 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.284 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.285 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.285 INFO:teuthology.orchestra.run.smithi012.stdout: "req_snapdiff_latency": { 2024-03-18T00:06:48.285 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.285 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.285 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.285 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.285 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmdir_latency": { 2024-03-18T00:06:48.285 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.285 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.285 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.285 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.286 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmsnap_latency": { 2024-03-18T00:06:48.286 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.286 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.286 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.286 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmxattr_latency": { 2024-03-18T00:06:48.286 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.286 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.286 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.286 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setattr_latency": { 2024-03-18T00:06:48.286 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.286 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setdirlayout_latency": { 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setfilelock_latency": { 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setlayout_latency": { 2024-03-18T00:06:48.287 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.288 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.288 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.288 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setxattr_latency": { 2024-03-18T00:06:48.288 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.288 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.288 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.289 INFO:teuthology.orchestra.run.smithi012.stdout: "req_symlink_latency": { 2024-03-18T00:06:48.289 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.289 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.289 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.289 INFO:teuthology.orchestra.run.smithi012.stdout: "req_unlink_latency": { 2024-03-18T00:06:48.289 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 661, 2024-03-18T00:06:48.289 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 1.753729228, 2024-03-18T00:06:48.289 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.002653145 2024-03-18T00:06:48.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.289 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_revoke_eviction": 0, 2024-03-18T00:06:48.289 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_acquisition_throttle": 0, 2024-03-18T00:06:48.290 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getvxattr_latency": { 2024-03-18T00:06:48.290 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.290 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.290 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.290 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:48.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.290 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_sessions": { 2024-03-18T00:06:48.290 INFO:teuthology.orchestra.run.smithi012.stdout: "session_count": 1, 2024-03-18T00:06:48.290 INFO:teuthology.orchestra.run.smithi012.stdout: "session_add": 2, 2024-03-18T00:06:48.290 INFO:teuthology.orchestra.run.smithi012.stdout: "session_remove": 1, 2024-03-18T00:06:48.290 INFO:teuthology.orchestra.run.smithi012.stdout: "sessions_open": 1, 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: "sessions_stale": 0, 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: "total_load": 1144, 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: "average_load": 1144, 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_session_uptime": 533, 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: "mdthresh_evicted": 0 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: "mempool": { 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_bytes": 0, 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_items": 0, 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_items": 0, 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_items": 0, 2024-03-18T00:06:48.291 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T00:06:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T00:06:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T00:06:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T00:06:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T00:06:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_items": 0, 2024-03-18T00:06:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T00:06:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T00:06:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_bytes": 0, 2024-03-18T00:06:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_items": 0, 2024-03-18T00:06:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_bytes": 0, 2024-03-18T00:06:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_items": 0, 2024-03-18T00:06:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T00:06:48.292 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_items": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_bytes": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_items": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_bytes": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_items": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_bytes": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_items": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T00:06:48.293 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_items": 0, 2024-03-18T00:06:48.294 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T00:06:48.294 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_items": 0, 2024-03-18T00:06:48.294 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_bytes": 299597, 2024-03-18T00:06:48.294 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_items": 128, 2024-03-18T00:06:48.294 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_bytes": 528, 2024-03-18T00:06:48.294 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_items": 6, 2024-03-18T00:06:48.294 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_bytes": 0, 2024-03-18T00:06:48.294 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_items": 0, 2024-03-18T00:06:48.294 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_bytes": 0, 2024-03-18T00:06:48.294 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_items": 0, 2024-03-18T00:06:48.294 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_bytes": 0, 2024-03-18T00:06:48.294 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_items": 0, 2024-03-18T00:06:48.294 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_bytes": 8792, 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_items": 168, 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_items": 0, 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_bytes": 0, 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_items": 0, 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_bytes": 132093, 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_items": 554, 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_bytes": 0, 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_items": 0, 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_bytes": 0, 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_items": 0 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "objecter": { 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "op_active": 0, 2024-03-18T00:06:48.295 INFO:teuthology.orchestra.run.smithi012.stdout: "op_laggy": 0, 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: "op_send": 3170, 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: "op_send_bytes": 9996809, 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: "op_resend": 29, 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: "op_reply": 3170, 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: "op_latency": { 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 3170, 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 82.762393234, 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.026108010 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: "op_inflight": 0, 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: "oplen_avg": { 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 3170, 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 3441 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.296 INFO:teuthology.orchestra.run.smithi012.stdout: "op": 3170, 2024-03-18T00:06:48.297 INFO:teuthology.orchestra.run.smithi012.stdout: "op_r": 74, 2024-03-18T00:06:48.297 INFO:teuthology.orchestra.run.smithi012.stdout: "op_w": 3096, 2024-03-18T00:06:48.297 INFO:teuthology.orchestra.run.smithi012.stdout: "op_rmw": 0, 2024-03-18T00:06:48.297 INFO:teuthology.orchestra.run.smithi012.stdout: "op_pg": 0, 2024-03-18T00:06:48.297 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_stat": 0, 2024-03-18T00:06:48.297 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_create": 10, 2024-03-18T00:06:48.297 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_read": 74, 2024-03-18T00:06:48.297 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_write": 420, 2024-03-18T00:06:48.297 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_writefull": 73, 2024-03-18T00:06:48.297 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_writesame": 0, 2024-03-18T00:06:48.297 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_append": 0, 2024-03-18T00:06:48.297 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_zero": 0, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_truncate": 4, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_delete": 2368, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_mapext": 0, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_sparse_read": 0, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_clonerange": 0, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_getxattr": 0, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_setxattr": 20, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_cmpxattr": 0, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_rmxattr": 0, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_resetxattrs": 0, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_call": 0, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_watch": 0, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_notify": 0, 2024-03-18T00:06:48.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_pgls": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_pgls_filter": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_other": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_active": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_send": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_resend": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_ping": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_active": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_send": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_resend": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_active": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_send": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_resend": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_active": 0, 2024-03-18T00:06:48.299 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_send": 0, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_resend": 0, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "command_active": 0, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "command_send": 0, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "command_resend": 0, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "map_epoch": 94, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "map_full": 0, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "map_inc": 4, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_sessions": 12, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_session_open": 12, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_session_close": 0, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_laggy": 0, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_wr": 394, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_rd": 0, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_del": 78 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.300 INFO:teuthology.orchestra.run.smithi012.stdout: "oft": { 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_objs": 1, 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_kv_pairs": 0, 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_updates": 0, 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_removes": 0 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "osd-slow-ops": { 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "slow_ops_count": 0 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "purge_queue": { 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_ops": 0, 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_ops_high_water": 10, 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing": 0, 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_high_water": 5, 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executed_ops": 2354, 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executed": 1177, 2024-03-18T00:06:48.301 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_item_in_journal": 0 2024-03-18T00:06:48.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.302 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T00:06:48.302 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-18T00:06:48.302 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-18T00:06:48.302 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-18T00:06:48.302 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 36966, 2024-03-18T00:06:48.302 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 11719080, 2024-03-18T00:06:48.302 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-18T00:06:48.302 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 36966, 2024-03-18T00:06:48.302 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-18T00:06:48.302 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-18T00:06:48.302 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 36966, 2024-03-18T00:06:48.302 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 11719080, 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-objecter_bytes": { 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-18T00:06:48.303 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 3170, 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 10125102, 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 3170, 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 10125102, 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-objecter_ops": { 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 1024, 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-18T00:06:48.304 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 3170, 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 3170, 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 3170, 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 3170, 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.305 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-write_buf_throttle": { 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 3758096384, 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 1177, 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 128293, 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 1177, 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 75, 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 128293, 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.306 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-write_buf_throttle-0x5654f6af2500": { 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 3758096384, 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 6234, 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 9031297, 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 6234, 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 345, 2024-03-18T00:06:48.307 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 9031297, 2024-03-18T00:06:48.308 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-18T00:06:48.308 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:48.308 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:48.308 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:48.308 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:48.308 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:48.308 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-18T00:06:49.176 INFO:tasks.check_counter:Daemon mds.a mds.exported=32 2024-03-18T00:06:49.176 INFO:tasks.check_counter:Daemon mds.a mds.imported=4 2024-03-18T00:06:49.177 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=629 2024-03-18T00:06:49.177 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=4296 2024-03-18T00:06:49.177 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=4 2024-03-18T00:06:49.177 DEBUG:tasks.check_counter:Getting stats from d 2024-03-18T00:06:49.177 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-18T00:06:49.685 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:06:49.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:49 smithi012 ceph-mon[31541]: pgmap v681: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 427 KiB/s wr, 274 op/s 2024-03-18T00:06:49.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:49 smithi191 ceph-mon[36708]: pgmap v681: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 427 KiB/s wr, 274 op/s 2024-03-18T00:06:50.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:49 smithi079 ceph-mon[36241]: pgmap v681: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 427 KiB/s wr, 274 op/s 2024-03-18T00:06:51.721 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:51 smithi012 ceph-mon[31541]: pgmap v682: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 416 KiB/s wr, 217 op/s 2024-03-18T00:06:51.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:51 smithi191 ceph-mon[36708]: pgmap v682: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 416 KiB/s wr, 217 op/s 2024-03-18T00:06:51.847 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:51 smithi079 ceph-mon[36241]: pgmap v682: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 416 KiB/s wr, 217 op/s 2024-03-18T00:06:52.067 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-18T00:06:52.067 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T00:06:52.067 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 727, 2024-03-18T00:06:52.067 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 703, 2024-03-18T00:06:52.067 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 164618663, 2024-03-18T00:06:52.067 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 281740, 2024-03-18T00:06:52.067 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 7, 2024-03-18T00:06:52.067 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 3, 2024-03-18T00:06:52.067 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.685926419, 2024-03-18T00:06:52.067 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.050185640, 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.552117440, 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.088434096, 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 702, 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.104715944, 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000149168 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 894208, 2024-03-18T00:06:52.068 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 124960 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 839, 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 991, 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 240796119, 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 601849, 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 7, 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 6, 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.801260954, 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.075314343, 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.601172179, 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.105916818, 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:06:52.069 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 989, 2024-03-18T00:06:52.070 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.156627553, 2024-03-18T00:06:52.070 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000158369 2024-03-18T00:06:52.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.070 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:06:52.070 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.070 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.070 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.070 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T00:06:52.070 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 335360 2024-03-18T00:06:52.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.070 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T00:06:52.070 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 1349, 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 1349, 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 189155014, 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 428216, 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 6, 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 5, 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.757548515, 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.091304282, 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.482379245, 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.164681452, 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 1348, 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.183055658, 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000135797 2024-03-18T00:06:52.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: "cct": { 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: "total_workers": 1, 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: "unhealthy_workers": 0 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: "finisher-MDSRank": { 2024-03-18T00:06:52.072 INFO:teuthology.orchestra.run.smithi012.stdout: "queue_len": 0, 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: "complete_latency": { 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 4159, 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.619370093, 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000148922 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: "finisher-PurgeQueue": { 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: "queue_len": 0, 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: "complete_latency": { 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.073 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": { 2024-03-18T00:06:52.074 INFO:teuthology.orchestra.run.smithi012.stdout: "request": 0, 2024-03-18T00:06:52.074 INFO:teuthology.orchestra.run.smithi012.stdout: "reply": 0, 2024-03-18T00:06:52.074 INFO:teuthology.orchestra.run.smithi012.stdout: "reply_latency": { 2024-03-18T00:06:52.074 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.074 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.074 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.074 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.074 INFO:teuthology.orchestra.run.smithi012.stdout: "slow_reply": 0, 2024-03-18T00:06:52.074 INFO:teuthology.orchestra.run.smithi012.stdout: "forward": 0, 2024-03-18T00:06:52.074 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_fetch_complete": 0, 2024-03-18T00:06:52.074 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_fetch_keys": 0, 2024-03-18T00:06:52.074 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_commit": 0, 2024-03-18T00:06:52.074 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_split": 0, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_merge": 0, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes": 48912, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_top": 11837, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_bottom": 37075, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_pin_tail": 0, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_pinned": 32001, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_expired": 0, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_with_caps": 0, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": 0, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "subtrees": 1, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse": 0, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_hit": 0, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_forward": 0, 2024-03-18T00:06:52.075 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_discover": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_dir_fetch": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_remote_ino": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "traverse_lock": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "load_cent": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "q": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "exported": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "exported_inodes": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "imported": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "imported_inodes": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_dir_fetch": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_backtrace_fetch": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "openino_peer_discover": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rfiles": 0, 2024-03-18T00:06:52.076 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rbytes": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "root_rsnaps": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_set_tag": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_inotable_repaired": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dir_inodes": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_file_inodes": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_inode_file_caps": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_grant": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T00:06:52.077 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_caps": 0, 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_caps_dirty": 0, 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_cap_release": 0, 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: "process_request_cap_release": 0 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_cache": { 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_update": 0, 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_update_receipt": 0, 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_try_discover": 0, 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_send_discover": 0, 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: "dir_handle_discover": 0, 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays": 0, 2024-03-18T00:06:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays_delayed": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "num_strays_enqueuing": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_created": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_enqueued": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_reintegrated": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "strays_migrated": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_processing": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_enqueued": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "num_recovering_prioritized": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "recovery_started": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "recovery_completed": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_quiesce_path": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_quiesce_inode": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_exportdir": 0, 2024-03-18T00:06:52.079 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_flush": 0, 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_fragmentdir": 0, 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_fragstats": 0, 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "ireq_inodestats": 0 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_log": { 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "evlrg": 0, 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "evadd": 0, 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "evex": 0, 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "evtrm": 0, 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "ev": 265375, 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "evexg": 0, 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "evexd": 0, 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "segadd": 0, 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "segex": 0, 2024-03-18T00:06:52.080 INFO:teuthology.orchestra.run.smithi012.stdout: "segtrm": 0, 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: "seg": 135, 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: "segmjr": 23, 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: "segexg": 0, 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: "segexd": 0, 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: "expos": 392639585, 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: "wrpos": 597083716, 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: "rdpos": 597082065, 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: "jlat": { 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: "replayed": 265375 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.081 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_mem": { 2024-03-18T00:06:52.082 INFO:teuthology.orchestra.run.smithi012.stdout: "ino": 15, 2024-03-18T00:06:52.082 INFO:teuthology.orchestra.run.smithi012.stdout: "ino+": 49441, 2024-03-18T00:06:52.082 INFO:teuthology.orchestra.run.smithi012.stdout: "ino-": 49426, 2024-03-18T00:06:52.082 INFO:teuthology.orchestra.run.smithi012.stdout: "dir": 23, 2024-03-18T00:06:52.082 INFO:teuthology.orchestra.run.smithi012.stdout: "dir+": 1555, 2024-03-18T00:06:52.082 INFO:teuthology.orchestra.run.smithi012.stdout: "dir-": 1532, 2024-03-18T00:06:52.082 INFO:teuthology.orchestra.run.smithi012.stdout: "dn": 48912, 2024-03-18T00:06:52.082 INFO:teuthology.orchestra.run.smithi012.stdout: "dn+": 109267, 2024-03-18T00:06:52.082 INFO:teuthology.orchestra.run.smithi012.stdout: "dn-": 60355, 2024-03-18T00:06:52.082 INFO:teuthology.orchestra.run.smithi012.stdout: "cap": 0, 2024-03-18T00:06:52.082 INFO:teuthology.orchestra.run.smithi012.stdout: "cap+": 0, 2024-03-18T00:06:52.082 INFO:teuthology.orchestra.run.smithi012.stdout: "cap-": 0, 2024-03-18T00:06:52.082 INFO:teuthology.orchestra.run.smithi012.stdout: "rss": 139972, 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: "heap": 215420 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_server": { 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: "dispatch_client_request": 0, 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: "dispatch_server_request": 0, 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_request": 0, 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_client_session": 0, 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: "handle_peer_request": 0, 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: "req_create_latency": { 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getattr_latency": { 2024-03-18T00:06:52.083 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.084 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.084 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.084 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getfilelock_latency": { 2024-03-18T00:06:52.084 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.084 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.084 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.084 INFO:teuthology.orchestra.run.smithi012.stdout: "req_link_latency": { 2024-03-18T00:06:52.084 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.084 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.084 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookup_latency": { 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookuphash_latency": { 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupino_latency": { 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.086 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupname_latency": { 2024-03-18T00:06:52.086 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.086 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.086 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.086 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupparent_latency": { 2024-03-18T00:06:52.086 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.086 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.086 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.086 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lookupsnap_latency": { 2024-03-18T00:06:52.086 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.086 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.087 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.087 INFO:teuthology.orchestra.run.smithi012.stdout: "req_lssnap_latency": { 2024-03-18T00:06:52.087 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.087 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.087 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.087 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mkdir_latency": { 2024-03-18T00:06:52.088 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.088 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.088 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.088 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mknod_latency": { 2024-03-18T00:06:52.088 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.088 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.088 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.088 INFO:teuthology.orchestra.run.smithi012.stdout: "req_mksnap_latency": { 2024-03-18T00:06:52.088 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.088 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.088 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: "req_open_latency": { 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: "req_readdir_latency": { 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rename_latency": { 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.089 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: "req_renamesnap_latency": { 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: "req_snapdiff_latency": { 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmdir_latency": { 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.090 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmsnap_latency": { 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: "req_rmxattr_latency": { 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setattr_latency": { 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.091 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setdirlayout_latency": { 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setfilelock_latency": { 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setlayout_latency": { 2024-03-18T00:06:52.092 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: "req_setxattr_latency": { 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: "req_symlink_latency": { 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: "req_unlink_latency": { 2024-03-18T00:06:52.093 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_revoke_eviction": 0, 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: "cap_acquisition_throttle": 0, 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: "req_getvxattr_latency": { 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_sessions": { 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: "session_count": 1, 2024-03-18T00:06:52.094 INFO:teuthology.orchestra.run.smithi012.stdout: "session_add": 2, 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: "session_remove": 1, 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: "sessions_open": 2, 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: "sessions_stale": 0, 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: "total_load": 0, 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: "average_load": 0, 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: "avg_session_uptime": 4061, 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: "mdthresh_evicted": 0 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: "mempool": { 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_bytes": 0, 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_items": 0, 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_items": 0, 2024-03-18T00:06:52.095 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_items": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_items": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_bytes": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_items": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_bytes": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_items": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T00:06:52.096 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_items": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_bytes": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_items": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_bytes": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_items": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_bytes": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_items": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T00:06:52.097 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_items": 0, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_items": 0, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_bytes": 240536, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_items": 81, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_bytes": 0, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_items": 0, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_bytes": 0, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_items": 0, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_bytes": 0, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_items": 0, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_bytes": 0, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_items": 0, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_bytes": 8792, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_items": 168, 2024-03-18T00:06:52.098 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_items": 0, 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_bytes": 0, 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_items": 0, 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_bytes": 34498603, 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_items": 196910, 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_bytes": 0, 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_items": 0, 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_bytes": 0, 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_items": 0 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: "objecter": { 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: "op_active": 0, 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: "op_laggy": 0, 2024-03-18T00:06:52.099 INFO:teuthology.orchestra.run.smithi012.stdout: "op_send": 2676, 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: "op_send_bytes": 21, 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: "op_resend": 3, 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: "op_reply": 2676, 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: "op_latency": { 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 2676, 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 23.779821317, 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.008886330 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: "op_inflight": 0, 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: "oplen_avg": { 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 2676, 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 2677 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: "op": 2676, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "op_r": 2676, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "op_w": 0, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "op_rmw": 0, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "op_pg": 0, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_stat": 1552, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_create": 0, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_read": 1122, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_write": 0, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_writefull": 0, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_writesame": 0, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_append": 0, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_zero": 0, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_truncate": 0, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_delete": 0, 2024-03-18T00:06:52.101 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_mapext": 0, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_sparse_read": 0, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_clonerange": 0, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_getxattr": 1, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_setxattr": 0, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_cmpxattr": 0, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_rmxattr": 0, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_resetxattrs": 0, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_call": 0, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_watch": 0, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_notify": 0, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_pgls": 0, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_pgls_filter": 0, 2024-03-18T00:06:52.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdop_other": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_active": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_send": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_resend": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "linger_ping": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_active": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_send": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "poolop_resend": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_active": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_send": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "poolstat_resend": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_active": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_send": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "statfs_resend": 0, 2024-03-18T00:06:52.103 INFO:teuthology.orchestra.run.smithi012.stdout: "command_active": 0, 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "command_send": 0, 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "command_resend": 0, 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "map_epoch": 94, 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "map_full": 0, 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "map_inc": 4, 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_sessions": 11, 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_session_open": 11, 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_session_close": 0, 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_laggy": 0, 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_wr": 0, 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_rd": 2, 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_del": 0 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "oft": { 2024-03-18T00:06:52.104 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_objs": 0, 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_kv_pairs": 0, 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_updates": 0, 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: "omap_total_removes": 0 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: "osd-slow-ops": { 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: "slow_ops_count": 0 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: "purge_queue": { 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_ops": 0, 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing": 0, 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executing_high_water": 0, 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executed_ops": 0, 2024-03-18T00:06:52.105 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_executed": 0, 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: "pq_item_in_journal": 0 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 2911, 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 594292827, 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 2911, 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 2911, 2024-03-18T00:06:52.106 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 594292827, 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-objecter_bytes": { 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-18T00:06:52.107 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 2676, 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 592937088, 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 2676, 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 592937088, 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-objecter_ops": { 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 1024, 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-18T00:06:52.108 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 2676, 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 2676, 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 2676, 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 2676, 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.109 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-write_buf_throttle": { 2024-03-18T00:06:52.110 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-18T00:06:52.110 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 3758096384, 2024-03-18T00:06:52.110 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-18T00:06:52.110 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-18T00:06:52.110 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-18T00:06:52.110 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-18T00:06:52.110 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-18T00:06:52.110 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-18T00:06:52.110 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-18T00:06:52.110 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 0, 2024-03-18T00:06:52.110 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 0, 2024-03-18T00:06:52.110 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-18T00:06:52.110 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-write_buf_throttle-0x5631628f2780": { 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 3758096384, 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 0, 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 0, 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 0, 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-18T00:06:52.111 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-18T00:06:52.112 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 0, 2024-03-18T00:06:52.112 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 0, 2024-03-18T00:06:52.112 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-18T00:06:52.112 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:52.112 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:52.112 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:52.112 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:52.112 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:52.112 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-18T00:06:53.063 INFO:tasks.check_counter:Daemon mds.d mds.exported=0 2024-03-18T00:06:53.064 INFO:tasks.check_counter:Daemon mds.d mds.imported=0 2024-03-18T00:06:53.064 INFO:tasks.check_counter:Daemon mds.d mds_cache.dir_update=0 2024-03-18T00:06:53.064 INFO:tasks.check_counter:Daemon mds.d mds_cache.dir_update_receipt=0 2024-03-18T00:06:53.064 INFO:tasks.check_counter:Daemon mds.d mds.dir_split=0 2024-03-18T00:06:53.064 DEBUG:tasks.check_counter:Getting stats from g 2024-03-18T00:06:53.064 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-18T00:06:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:06:53.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:53 smithi191 ceph-mon[36708]: pgmap v683: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 30 KiB/s wr, 87 op/s 2024-03-18T00:06:53.881 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:53 smithi012 ceph-mon[31541]: pgmap v683: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 30 KiB/s wr, 87 op/s 2024-03-18T00:06:54.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:53 smithi079 ceph-mon[36241]: pgmap v683: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 30 KiB/s wr, 87 op/s 2024-03-18T00:06:55.791 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-18T00:06:55.792 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T00:06:55.792 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 0, 2024-03-18T00:06:55.792 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 2, 2024-03-18T00:06:55.792 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 0, 2024-03-18T00:06:55.792 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 0, 2024-03-18T00:06:55.792 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 4, 2024-03-18T00:06:55.792 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 0, 2024-03-18T00:06:55.792 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.004496514, 2024-03-18T00:06:55.792 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T00:06:55.792 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.000729243, 2024-03-18T00:06:55.792 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T00:06:55.792 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T00:06:55.793 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 1, 2024-03-18T00:06:55.794 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 150, 2024-03-18T00:06:55.794 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 160, 2024-03-18T00:06:55.794 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 104160, 2024-03-18T00:06:55.794 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 3, 2024-03-18T00:06:55.794 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 2, 2024-03-18T00:06:55.794 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.030589207, 2024-03-18T00:06:55.794 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.025597399, 2024-03-18T00:06:55.794 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.002618122, 2024-03-18T00:06:55.794 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T00:06:55.794 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:06:55.794 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 148, 2024-03-18T00:06:55.794 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.018451245, 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000124670 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 104160 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 207, 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 200, 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 749696, 2024-03-18T00:06:55.795 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 121952, 2024-03-18T00:06:55.796 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 2, 2024-03-18T00:06:55.796 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 1, 2024-03-18T00:06:55.796 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.068092806, 2024-03-18T00:06:55.796 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.028799827, 2024-03-18T00:06:55.796 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.032981990, 2024-03-18T00:06:55.796 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.009653792, 2024-03-18T00:06:55.796 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:06:55.796 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 200, 2024-03-18T00:06:55.796 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.026877874, 2024-03-18T00:06:55.796 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000134389 2024-03-18T00:06:55.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:55.796 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:06:55.796 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 749696, 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 121952 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: "cct": { 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: "total_workers": 0, 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: "unhealthy_workers": 0 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: "mempool": { 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_bytes": 0, 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_items": 0, 2024-03-18T00:06:55.797 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_items": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_items": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_items": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_bytes": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_items": 0, 2024-03-18T00:06:55.798 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_bytes": 0, 2024-03-18T00:06:55.799 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_items": 0, 2024-03-18T00:06:55.799 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T00:06:55.799 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T00:06:55.799 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T00:06:55.799 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T00:06:55.799 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T00:06:55.799 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_items": 0, 2024-03-18T00:06:55.799 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_bytes": 0, 2024-03-18T00:06:55.799 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_items": 0, 2024-03-18T00:06:55.799 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T00:06:55.799 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T00:06:55.799 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_bytes": 0, 2024-03-18T00:06:55.799 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_items": 0, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_bytes": 0, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_items": 0, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_items": 0, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_items": 0, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_bytes": 151029, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_items": 31, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_bytes": 0, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_items": 0, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_bytes": 0, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_items": 0, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_bytes": 0, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_items": 0, 2024-03-18T00:06:55.800 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_bytes": 0, 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_items": 0, 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_bytes": 0, 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_items": 0, 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_items": 0, 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_bytes": 0, 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_items": 0, 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_bytes": 928, 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_items": 2, 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_bytes": 0, 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_items": 0, 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_bytes": 0, 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_items": 0 2024-03-18T00:06:55.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:55.802 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T00:06:55.802 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-18T00:06:55.802 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-18T00:06:55.802 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-18T00:06:55.802 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 204, 2024-03-18T00:06:55.802 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 717976, 2024-03-18T00:06:55.802 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-18T00:06:55.802 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 204, 2024-03-18T00:06:55.802 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-18T00:06:55.802 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-18T00:06:55.802 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 204, 2024-03-18T00:06:55.802 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 717976, 2024-03-18T00:06:55.802 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-18T00:06:55.803 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:55.803 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:55.803 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:55.803 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:55.803 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:55.803 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-18T00:06:55.812 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:55 smithi012 ceph-mon[31541]: pgmap v684: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 21 KiB/s wr, 55 op/s 2024-03-18T00:06:55.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:55 smithi191 ceph-mon[36708]: pgmap v684: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 21 KiB/s wr, 55 op/s 2024-03-18T00:06:56.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:55 smithi079 ceph-mon[36241]: pgmap v684: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 21 KiB/s wr, 55 op/s 2024-03-18T00:06:56.680 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-18T00:06:56.680 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-18T00:06:56.680 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-18T00:06:56.680 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-18T00:06:56.680 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-03-18T00:06:56.680 DEBUG:tasks.check_counter:Getting stats from j 2024-03-18T00:06:56.681 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-18T00:06:57.171 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:06:57.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:57 smithi191 ceph-mon[36708]: pgmap v685: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 12 KiB/s wr, 29 op/s 2024-03-18T00:06:57.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:57 smithi012 ceph-mon[31541]: pgmap v685: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 12 KiB/s wr, 29 op/s 2024-03-18T00:06:58.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:57 smithi079 ceph-mon[36241]: pgmap v685: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 12 KiB/s wr, 29 op/s 2024-03-18T00:06:59.367 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-18T00:06:59.367 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T00:06:59.367 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 0, 2024-03-18T00:06:59.367 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 2, 2024-03-18T00:06:59.367 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 0, 2024-03-18T00:06:59.367 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 0, 2024-03-18T00:06:59.367 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 4, 2024-03-18T00:06:59.367 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 0, 2024-03-18T00:06:59.367 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.005342806, 2024-03-18T00:06:59.367 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T00:06:59.368 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.000901909, 2024-03-18T00:06:59.368 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T00:06:59.368 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:06:59.368 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:59.368 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:59.368 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:59.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:59.368 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:06:59.368 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:59.368 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:59.368 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:59.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:59.368 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T00:06:59.369 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T00:06:59.369 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:59.369 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T00:06:59.369 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 213, 2024-03-18T00:06:59.369 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 206, 2024-03-18T00:06:59.369 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 751328, 2024-03-18T00:06:59.369 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 125792, 2024-03-18T00:06:59.369 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 2, 2024-03-18T00:06:59.369 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 1, 2024-03-18T00:06:59.369 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.078280770, 2024-03-18T00:06:59.369 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.036363096, 2024-03-18T00:06:59.369 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.038920719, 2024-03-18T00:06:59.369 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.008135174, 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 206, 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.024283278, 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000117879 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 751328, 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 125792 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:59.370 INFO:teuthology.orchestra.run.smithi012.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_messages": 1, 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages": 155, 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_bytes": 160, 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_bytes": 105040, 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_created_connections": 3, 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_active_connections": 2, 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_total_time": 0.025072608, 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_send_time": 0.020182599, 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_recv_time": 0.002234164, 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 153, 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.017309096, 2024-03-18T00:06:59.371 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000113131 2024-03-18T00:06:59.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:59.372 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:06:59.372 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:59.372 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:59.372 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:59.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:59.372 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T00:06:59.372 INFO:teuthology.orchestra.run.smithi012.stdout: "msgr_send_encrypted_bytes": 105040 2024-03-18T00:06:59.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:59.372 INFO:teuthology.orchestra.run.smithi012.stdout: "cct": { 2024-03-18T00:06:59.372 INFO:teuthology.orchestra.run.smithi012.stdout: "total_workers": 0, 2024-03-18T00:06:59.372 INFO:teuthology.orchestra.run.smithi012.stdout: "unhealthy_workers": 0 2024-03-18T00:06:59.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:59.373 INFO:teuthology.orchestra.run.smithi012.stdout: "mempool": { 2024-03-18T00:06:59.373 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_bytes": 0, 2024-03-18T00:06:59.373 INFO:teuthology.orchestra.run.smithi012.stdout: "bloom_filter_items": 0, 2024-03-18T00:06:59.373 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T00:06:59.373 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_alloc_items": 0, 2024-03-18T00:06:59.373 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T00:06:59.373 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_data_items": 0, 2024-03-18T00:06:59.373 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T00:06:59.373 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T00:06:59.373 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T00:06:59.373 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T00:06:59.373 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T00:06:59.373 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_other_items": 0, 2024-03-18T00:06:59.374 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T00:06:59.374 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T00:06:59.374 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_bytes": 0, 2024-03-18T00:06:59.374 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_extent_items": 0, 2024-03-18T00:06:59.374 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_bytes": 0, 2024-03-18T00:06:59.374 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_blob_items": 0, 2024-03-18T00:06:59.374 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T00:06:59.374 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T00:06:59.374 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T00:06:59.374 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T00:06:59.374 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T00:06:59.374 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_fsck_items": 0, 2024-03-18T00:06:59.374 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_bytes": 0, 2024-03-18T00:06:59.375 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_txc_items": 0, 2024-03-18T00:06:59.375 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T00:06:59.375 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T00:06:59.375 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_bytes": 0, 2024-03-18T00:06:59.375 INFO:teuthology.orchestra.run.smithi012.stdout: "bluestore_writing_items": 0, 2024-03-18T00:06:59.375 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_bytes": 0, 2024-03-18T00:06:59.375 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_items": 0, 2024-03-18T00:06:59.375 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T00:06:59.375 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_reader_items": 0, 2024-03-18T00:06:59.375 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T00:06:59.375 INFO:teuthology.orchestra.run.smithi012.stdout: "bluefs_file_writer_items": 0, 2024-03-18T00:06:59.375 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_bytes": 148967, 2024-03-18T00:06:59.375 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_anon_items": 29, 2024-03-18T00:06:59.376 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_bytes": 0, 2024-03-18T00:06:59.376 INFO:teuthology.orchestra.run.smithi012.stdout: "buffer_meta_items": 0, 2024-03-18T00:06:59.376 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_bytes": 0, 2024-03-18T00:06:59.376 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_items": 0, 2024-03-18T00:06:59.376 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_bytes": 0, 2024-03-18T00:06:59.376 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_mapbl_items": 0, 2024-03-18T00:06:59.376 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_bytes": 0, 2024-03-18T00:06:59.376 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_pglog_items": 0, 2024-03-18T00:06:59.376 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_bytes": 0, 2024-03-18T00:06:59.376 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_items": 0, 2024-03-18T00:06:59.376 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T00:06:59.376 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_mapping_items": 0, 2024-03-18T00:06:59.376 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_bytes": 0, 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "pgmap_items": 0, 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_bytes": 928, 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_co_items": 2, 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_bytes": 0, 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_1_items": 0, 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_bytes": 0, 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "unittest_2_items": 0 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 0, 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "max": 104857600, 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "get_started": 0, 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "get": 210, 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "get_sum": 718756, 2024-03-18T00:06:59.377 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_fail": 0, 2024-03-18T00:06:59.378 INFO:teuthology.orchestra.run.smithi012.stdout: "get_or_fail_success": 210, 2024-03-18T00:06:59.378 INFO:teuthology.orchestra.run.smithi012.stdout: "take": 0, 2024-03-18T00:06:59.378 INFO:teuthology.orchestra.run.smithi012.stdout: "take_sum": 0, 2024-03-18T00:06:59.378 INFO:teuthology.orchestra.run.smithi012.stdout: "put": 210, 2024-03-18T00:06:59.378 INFO:teuthology.orchestra.run.smithi012.stdout: "put_sum": 718756, 2024-03-18T00:06:59.378 INFO:teuthology.orchestra.run.smithi012.stdout: "wait": { 2024-03-18T00:06:59.378 INFO:teuthology.orchestra.run.smithi012.stdout: "avgcount": 0, 2024-03-18T00:06:59.378 INFO:teuthology.orchestra.run.smithi012.stdout: "sum": 0.000000000, 2024-03-18T00:06:59.378 INFO:teuthology.orchestra.run.smithi012.stdout: "avgtime": 0.000000000 2024-03-18T00:06:59.378 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:59.378 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-03-18T00:06:59.378 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-18T00:06:59.669 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:06:59 smithi012 ceph-mon[31541]: pgmap v686: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-03-18T00:07:00.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:06:59 smithi079 ceph-mon[36241]: pgmap v686: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-03-18T00:07:00.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:06:59 smithi191 ceph-mon[36708]: pgmap v686: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-03-18T00:07:00.362 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-18T00:07:00.362 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-18T00:07:00.362 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-18T00:07:00.362 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-18T00:07:00.362 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-18T00:07:00.362 DEBUG:tasks.check_counter:Getting stats from b 2024-03-18T00:07:00.362 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-18T00:07:00.858 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T00:07:01.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:01 smithi012 ceph-mon[31541]: pgmap v687: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.5 KiB/s wr, 12 op/s 2024-03-18T00:07:02.028 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:01 smithi079 ceph-mon[36241]: pgmap v687: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.5 KiB/s wr, 12 op/s 2024-03-18T00:07:02.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:01 smithi191 ceph-mon[36708]: pgmap v687: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.5 KiB/s wr, 12 op/s 2024-03-18T00:07:02.907 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-03-18T00:07:02.907 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T00:07:02.907 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 3333, 2024-03-18T00:07:02.907 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 3526, 2024-03-18T00:07:02.908 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 1990278, 2024-03-18T00:07:02.908 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 5303493, 2024-03-18T00:07:02.908 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 9, 2024-03-18T00:07:02.908 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 3, 2024-03-18T00:07:02.908 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.233857123, 2024-03-18T00:07:02.908 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.080554657, 2024-03-18T00:07:02.908 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.111161149, 2024-03-18T00:07:02.908 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.048176242, 2024-03-18T00:07:02.908 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:02.908 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 3531, 2024-03-18T00:07:02.909 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.363156267, 2024-03-18T00:07:02.909 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000102847 2024-03-18T00:07:02.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.909 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:02.909 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 4303, 2024-03-18T00:07:02.909 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000811775, 2024-03-18T00:07:02.909 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000188 2024-03-18T00:07:02.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.909 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 933952, 2024-03-18T00:07:02.910 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 130224 2024-03-18T00:07:02.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.910 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T00:07:02.910 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 16226, 2024-03-18T00:07:02.910 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 12880, 2024-03-18T00:07:02.910 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 7118519, 2024-03-18T00:07:02.910 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 3869847, 2024-03-18T00:07:02.910 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 10, 2024-03-18T00:07:02.910 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 6, 2024-03-18T00:07:02.910 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.999699375, 2024-03-18T00:07:02.911 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.432482016, 2024-03-18T00:07:02.911 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.679176667, 2024-03-18T00:07:02.911 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.038613073, 2024-03-18T00:07:02.911 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:02.911 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 12881, 2024-03-18T00:07:02.911 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.629397993, 2024-03-18T00:07:02.911 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000048862 2024-03-18T00:07:02.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.911 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:02.911 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 26092, 2024-03-18T00:07:02.911 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.004499009, 2024-03-18T00:07:02.911 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000172 2024-03-18T00:07:02.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.912 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T00:07:02.912 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 428384 2024-03-18T00:07:02.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.912 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T00:07:02.912 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 20962, 2024-03-18T00:07:02.912 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 24468, 2024-03-18T00:07:02.912 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 7002568, 2024-03-18T00:07:02.912 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 20268182, 2024-03-18T00:07:02.912 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 20, 2024-03-18T00:07:02.912 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 13, 2024-03-18T00:07:02.913 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 1.161242270, 2024-03-18T00:07:02.913 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.473393712, 2024-03-18T00:07:02.913 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.764682386, 2024-03-18T00:07:02.913 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.071782364, 2024-03-18T00:07:02.913 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:02.913 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 24458, 2024-03-18T00:07:02.913 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.962258282, 2024-03-18T00:07:02.913 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000039343 2024-03-18T00:07:02.913 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.913 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:02.913 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 37846, 2024-03-18T00:07:02.914 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.006550365, 2024-03-18T00:07:02.914 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000173 2024-03-18T00:07:02.914 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.914 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 89296, 2024-03-18T00:07:02.914 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 32320 2024-03-18T00:07:02.914 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.914 INFO:teuthology.orchestra.run.smithi079.stdout: "cct": { 2024-03-18T00:07:02.914 INFO:teuthology.orchestra.run.smithi079.stdout: "total_workers": 1, 2024-03-18T00:07:02.914 INFO:teuthology.orchestra.run.smithi079.stdout: "unhealthy_workers": 0 2024-03-18T00:07:02.914 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.915 INFO:teuthology.orchestra.run.smithi079.stdout: "finisher-MDSRank": { 2024-03-18T00:07:02.915 INFO:teuthology.orchestra.run.smithi079.stdout: "queue_len": 0, 2024-03-18T00:07:02.915 INFO:teuthology.orchestra.run.smithi079.stdout: "complete_latency": { 2024-03-18T00:07:02.915 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 1100, 2024-03-18T00:07:02.915 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 2.624211562, 2024-03-18T00:07:02.915 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.002385646 2024-03-18T00:07:02.915 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:02.915 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.915 INFO:teuthology.orchestra.run.smithi079.stdout: "finisher-PurgeQueue": { 2024-03-18T00:07:02.915 INFO:teuthology.orchestra.run.smithi079.stdout: "queue_len": 0, 2024-03-18T00:07:02.915 INFO:teuthology.orchestra.run.smithi079.stdout: "complete_latency": { 2024-03-18T00:07:02.916 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 2142, 2024-03-18T00:07:02.916 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.171084194, 2024-03-18T00:07:02.916 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000079871 2024-03-18T00:07:02.916 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:02.916 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.916 INFO:teuthology.orchestra.run.smithi079.stdout: "mds": { 2024-03-18T00:07:02.916 INFO:teuthology.orchestra.run.smithi079.stdout: "request": 13333, 2024-03-18T00:07:02.916 INFO:teuthology.orchestra.run.smithi079.stdout: "reply": 12785, 2024-03-18T00:07:02.916 INFO:teuthology.orchestra.run.smithi079.stdout: "reply_latency": { 2024-03-18T00:07:02.916 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 12785, 2024-03-18T00:07:02.916 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 579.671222918, 2024-03-18T00:07:02.916 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.045339947 2024-03-18T00:07:02.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.917 INFO:teuthology.orchestra.run.smithi079.stdout: "slow_reply": 0, 2024-03-18T00:07:02.917 INFO:teuthology.orchestra.run.smithi079.stdout: "forward": 548, 2024-03-18T00:07:02.917 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_fetch_complete": 0, 2024-03-18T00:07:02.917 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_fetch_keys": 0, 2024-03-18T00:07:02.917 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_commit": 30, 2024-03-18T00:07:02.917 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_split": 2, 2024-03-18T00:07:02.917 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_merge": 0, 2024-03-18T00:07:02.917 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes": 33, 2024-03-18T00:07:02.917 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_top": 7, 2024-03-18T00:07:02.917 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_bottom": 8, 2024-03-18T00:07:02.918 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_pin_tail": 18, 2024-03-18T00:07:02.918 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_pinned": 23, 2024-03-18T00:07:02.918 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_expired": 680, 2024-03-18T00:07:02.918 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_with_caps": 1, 2024-03-18T00:07:02.918 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": 1, 2024-03-18T00:07:02.918 INFO:teuthology.orchestra.run.smithi079.stdout: "subtrees": 3, 2024-03-18T00:07:02.918 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse": 29000, 2024-03-18T00:07:02.918 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_hit": 25960, 2024-03-18T00:07:02.918 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_forward": 548, 2024-03-18T00:07:02.918 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_discover": 239, 2024-03-18T00:07:02.918 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_dir_fetch": 0, 2024-03-18T00:07:02.919 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_remote_ino": 0, 2024-03-18T00:07:02.919 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_lock": 81, 2024-03-18T00:07:02.919 INFO:teuthology.orchestra.run.smithi079.stdout: "load_cent": 0, 2024-03-18T00:07:02.919 INFO:teuthology.orchestra.run.smithi079.stdout: "q": 0, 2024-03-18T00:07:02.919 INFO:teuthology.orchestra.run.smithi079.stdout: "exported": 17, 2024-03-18T00:07:02.919 INFO:teuthology.orchestra.run.smithi079.stdout: "exported_inodes": 218, 2024-03-18T00:07:02.919 INFO:teuthology.orchestra.run.smithi079.stdout: "imported": 3, 2024-03-18T00:07:02.919 INFO:teuthology.orchestra.run.smithi079.stdout: "imported_inodes": 483, 2024-03-18T00:07:02.919 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_dir_fetch": 0, 2024-03-18T00:07:02.919 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_backtrace_fetch": 0, 2024-03-18T00:07:02.919 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_peer_discover": 0, 2024-03-18T00:07:02.919 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rfiles": 4, 2024-03-18T00:07:02.920 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rbytes": 234, 2024-03-18T00:07:02.920 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rsnaps": 0, 2024-03-18T00:07:02.920 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T00:07:02.920 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_set_tag": 0, 2024-03-18T00:07:02.920 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T00:07:02.920 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_inotable_repaired": 0, 2024-03-18T00:07:02.920 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dir_inodes": 0, 2024-03-18T00:07:02.920 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T00:07:02.920 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T00:07:02.920 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_file_inodes": 0, 2024-03-18T00:07:02.920 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_inode_file_caps": 45, 2024-03-18T00:07:02.920 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_revoke": 631, 2024-03-18T00:07:02.920 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_grant": 1981, 2024-03-18T00:07:02.921 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_trunc": 5, 2024-03-18T00:07:02.921 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T00:07:02.921 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T00:07:02.921 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_caps": 1644, 2024-03-18T00:07:02.921 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_caps_dirty": 773, 2024-03-18T00:07:02.921 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_cap_release": 58, 2024-03-18T00:07:02.921 INFO:teuthology.orchestra.run.smithi079.stdout: "process_request_cap_release": 2825 2024-03-18T00:07:02.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.921 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache": { 2024-03-18T00:07:02.921 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_update": 353, 2024-03-18T00:07:02.921 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_update_receipt": 4859, 2024-03-18T00:07:02.921 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_try_discover": 3118, 2024-03-18T00:07:02.922 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_send_discover": 3244, 2024-03-18T00:07:02.922 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_handle_discover": 157, 2024-03-18T00:07:02.922 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays": 0, 2024-03-18T00:07:02.922 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays_delayed": 0, 2024-03-18T00:07:02.922 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays_enqueuing": 0, 2024-03-18T00:07:02.922 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_created": 1003, 2024-03-18T00:07:02.922 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_enqueued": 1003, 2024-03-18T00:07:02.922 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_reintegrated": 0, 2024-03-18T00:07:02.922 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_migrated": 0, 2024-03-18T00:07:02.922 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_processing": 0, 2024-03-18T00:07:02.922 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_enqueued": 0, 2024-03-18T00:07:02.922 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_prioritized": 0, 2024-03-18T00:07:02.923 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_started": 0, 2024-03-18T00:07:02.923 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_completed": 0, 2024-03-18T00:07:02.923 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_quiesce_path": 0, 2024-03-18T00:07:02.923 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_quiesce_inode": 0, 2024-03-18T00:07:02.923 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T00:07:02.923 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_exportdir": 17, 2024-03-18T00:07:02.923 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_flush": 0, 2024-03-18T00:07:02.923 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_fragmentdir": 2, 2024-03-18T00:07:02.923 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_fragstats": 0, 2024-03-18T00:07:02.923 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_inodestats": 0 2024-03-18T00:07:02.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.923 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log": { 2024-03-18T00:07:02.923 INFO:teuthology.orchestra.run.smithi079.stdout: "evlrg": 0, 2024-03-18T00:07:02.924 INFO:teuthology.orchestra.run.smithi079.stdout: "evadd": 4872, 2024-03-18T00:07:02.924 INFO:teuthology.orchestra.run.smithi079.stdout: "evex": 0, 2024-03-18T00:07:02.924 INFO:teuthology.orchestra.run.smithi079.stdout: "evtrm": 0, 2024-03-18T00:07:02.924 INFO:teuthology.orchestra.run.smithi079.stdout: "ev": 4872, 2024-03-18T00:07:02.924 INFO:teuthology.orchestra.run.smithi079.stdout: "evexg": 0, 2024-03-18T00:07:02.924 INFO:teuthology.orchestra.run.smithi079.stdout: "evexd": 0, 2024-03-18T00:07:02.924 INFO:teuthology.orchestra.run.smithi079.stdout: "segadd": 8, 2024-03-18T00:07:02.924 INFO:teuthology.orchestra.run.smithi079.stdout: "segex": 0, 2024-03-18T00:07:02.924 INFO:teuthology.orchestra.run.smithi079.stdout: "segtrm": 0, 2024-03-18T00:07:02.924 INFO:teuthology.orchestra.run.smithi079.stdout: "seg": 8, 2024-03-18T00:07:02.924 INFO:teuthology.orchestra.run.smithi079.stdout: "segmjr": 2, 2024-03-18T00:07:02.924 INFO:teuthology.orchestra.run.smithi079.stdout: "segexg": 0, 2024-03-18T00:07:02.924 INFO:teuthology.orchestra.run.smithi079.stdout: "segexd": 0, 2024-03-18T00:07:02.925 INFO:teuthology.orchestra.run.smithi079.stdout: "expos": 4194304, 2024-03-18T00:07:02.925 INFO:teuthology.orchestra.run.smithi079.stdout: "wrpos": 12643138, 2024-03-18T00:07:02.925 INFO:teuthology.orchestra.run.smithi079.stdout: "rdpos": 0, 2024-03-18T00:07:02.925 INFO:teuthology.orchestra.run.smithi079.stdout: "jlat": { 2024-03-18T00:07:02.925 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 418, 2024-03-18T00:07:02.925 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 1.963810057, 2024-03-18T00:07:02.925 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.004698110 2024-03-18T00:07:02.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.925 INFO:teuthology.orchestra.run.smithi079.stdout: "replayed": 0 2024-03-18T00:07:02.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.925 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_mem": { 2024-03-18T00:07:02.925 INFO:teuthology.orchestra.run.smithi079.stdout: "ino": 34, 2024-03-18T00:07:02.926 INFO:teuthology.orchestra.run.smithi079.stdout: "ino+": 1301, 2024-03-18T00:07:02.926 INFO:teuthology.orchestra.run.smithi079.stdout: "ino-": 1267, 2024-03-18T00:07:02.926 INFO:teuthology.orchestra.run.smithi079.stdout: "dir": 29, 2024-03-18T00:07:02.926 INFO:teuthology.orchestra.run.smithi079.stdout: "dir+": 1662, 2024-03-18T00:07:02.926 INFO:teuthology.orchestra.run.smithi079.stdout: "dir-": 1633, 2024-03-18T00:07:02.926 INFO:teuthology.orchestra.run.smithi079.stdout: "dn": 33, 2024-03-18T00:07:02.926 INFO:teuthology.orchestra.run.smithi079.stdout: "dn+": 2523, 2024-03-18T00:07:02.926 INFO:teuthology.orchestra.run.smithi079.stdout: "dn-": 2490, 2024-03-18T00:07:02.926 INFO:teuthology.orchestra.run.smithi079.stdout: "cap": 1, 2024-03-18T00:07:02.926 INFO:teuthology.orchestra.run.smithi079.stdout: "cap+": 1488, 2024-03-18T00:07:02.927 INFO:teuthology.orchestra.run.smithi079.stdout: "cap-": 1487, 2024-03-18T00:07:02.927 INFO:teuthology.orchestra.run.smithi079.stdout: "rss": 65548, 2024-03-18T00:07:02.927 INFO:teuthology.orchestra.run.smithi079.stdout: "heap": 231804 2024-03-18T00:07:02.927 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.927 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_server": { 2024-03-18T00:07:02.927 INFO:teuthology.orchestra.run.smithi079.stdout: "dispatch_client_request": 15305, 2024-03-18T00:07:02.927 INFO:teuthology.orchestra.run.smithi079.stdout: "dispatch_server_request": 91, 2024-03-18T00:07:02.927 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_request": 13333, 2024-03-18T00:07:02.927 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_session": 24, 2024-03-18T00:07:02.927 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_peer_request": 1514, 2024-03-18T00:07:02.927 INFO:teuthology.orchestra.run.smithi079.stdout: "req_create_latency": { 2024-03-18T00:07:02.927 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 792, 2024-03-18T00:07:02.927 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 6.109197984, 2024-03-18T00:07:02.928 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.007713633 2024-03-18T00:07:02.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.928 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getattr_latency": { 2024-03-18T00:07:02.928 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 250, 2024-03-18T00:07:02.928 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 1.736805540, 2024-03-18T00:07:02.928 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.006947222 2024-03-18T00:07:02.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.928 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getfilelock_latency": { 2024-03-18T00:07:02.928 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.928 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.928 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.929 INFO:teuthology.orchestra.run.smithi079.stdout: "req_link_latency": { 2024-03-18T00:07:02.929 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.929 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.929 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.929 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookup_latency": { 2024-03-18T00:07:02.929 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 10061, 2024-03-18T00:07:02.929 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 564.077228276, 2024-03-18T00:07:02.929 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.056065721 2024-03-18T00:07:02.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.929 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookuphash_latency": { 2024-03-18T00:07:02.929 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.930 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.930 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.930 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupino_latency": { 2024-03-18T00:07:02.930 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.930 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.930 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.930 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupname_latency": { 2024-03-18T00:07:02.930 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.930 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.931 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.931 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupparent_latency": { 2024-03-18T00:07:02.931 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.931 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.931 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.931 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupsnap_latency": { 2024-03-18T00:07:02.931 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.931 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.931 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.932 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lssnap_latency": { 2024-03-18T00:07:02.932 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.932 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.933 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.933 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mkdir_latency": { 2024-03-18T00:07:02.933 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 16, 2024-03-18T00:07:02.933 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.019110116, 2024-03-18T00:07:02.933 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.001194382 2024-03-18T00:07:02.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.933 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mknod_latency": { 2024-03-18T00:07:02.933 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.933 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.933 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.934 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mksnap_latency": { 2024-03-18T00:07:02.934 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.934 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.934 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.934 INFO:teuthology.orchestra.run.smithi079.stdout: "req_open_latency": { 2024-03-18T00:07:02.934 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 332, 2024-03-18T00:07:02.934 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.698529136, 2024-03-18T00:07:02.934 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.002104003 2024-03-18T00:07:02.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.935 INFO:teuthology.orchestra.run.smithi079.stdout: "req_readdir_latency": { 2024-03-18T00:07:02.935 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 81, 2024-03-18T00:07:02.935 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.631595606, 2024-03-18T00:07:02.935 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.007797476 2024-03-18T00:07:02.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.935 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rename_latency": { 2024-03-18T00:07:02.935 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 435, 2024-03-18T00:07:02.935 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 4.359224796, 2024-03-18T00:07:02.935 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.010021206 2024-03-18T00:07:02.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.935 INFO:teuthology.orchestra.run.smithi079.stdout: "req_renamesnap_latency": { 2024-03-18T00:07:02.936 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.936 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.936 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.936 INFO:teuthology.orchestra.run.smithi079.stdout: "req_snapdiff_latency": { 2024-03-18T00:07:02.936 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.936 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.936 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.936 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmdir_latency": { 2024-03-18T00:07:02.936 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 10, 2024-03-18T00:07:02.936 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.189715672, 2024-03-18T00:07:02.937 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.018971567 2024-03-18T00:07:02.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.937 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmsnap_latency": { 2024-03-18T00:07:02.937 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.937 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.937 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.937 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmxattr_latency": { 2024-03-18T00:07:02.937 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.937 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.937 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.938 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setattr_latency": { 2024-03-18T00:07:02.938 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 7, 2024-03-18T00:07:02.938 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.034796879, 2024-03-18T00:07:02.938 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.004970982 2024-03-18T00:07:02.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.938 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setdirlayout_latency": { 2024-03-18T00:07:02.938 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.938 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.938 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.938 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setfilelock_latency": { 2024-03-18T00:07:02.938 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.939 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.939 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.939 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setlayout_latency": { 2024-03-18T00:07:02.939 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.939 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.939 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.939 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setxattr_latency": { 2024-03-18T00:07:02.939 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 9, 2024-03-18T00:07:02.939 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.021292054, 2024-03-18T00:07:02.940 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.002365783 2024-03-18T00:07:02.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.940 INFO:teuthology.orchestra.run.smithi079.stdout: "req_symlink_latency": { 2024-03-18T00:07:02.940 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 3, 2024-03-18T00:07:02.940 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.002382999, 2024-03-18T00:07:02.940 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000794333 2024-03-18T00:07:02.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.940 INFO:teuthology.orchestra.run.smithi079.stdout: "req_unlink_latency": { 2024-03-18T00:07:02.940 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 789, 2024-03-18T00:07:02.940 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 1.791343860, 2024-03-18T00:07:02.940 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.002270397 2024-03-18T00:07:02.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.941 INFO:teuthology.orchestra.run.smithi079.stdout: "cap_revoke_eviction": 0, 2024-03-18T00:07:02.941 INFO:teuthology.orchestra.run.smithi079.stdout: "cap_acquisition_throttle": 0, 2024-03-18T00:07:02.941 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getvxattr_latency": { 2024-03-18T00:07:02.941 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.941 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.941 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.941 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:02.941 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.941 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_sessions": { 2024-03-18T00:07:02.941 INFO:teuthology.orchestra.run.smithi079.stdout: "session_count": 1, 2024-03-18T00:07:02.941 INFO:teuthology.orchestra.run.smithi079.stdout: "session_add": 4, 2024-03-18T00:07:02.941 INFO:teuthology.orchestra.run.smithi079.stdout: "session_remove": 3, 2024-03-18T00:07:02.941 INFO:teuthology.orchestra.run.smithi079.stdout: "sessions_open": 1, 2024-03-18T00:07:02.942 INFO:teuthology.orchestra.run.smithi079.stdout: "sessions_stale": 0, 2024-03-18T00:07:02.942 INFO:teuthology.orchestra.run.smithi079.stdout: "total_load": 699, 2024-03-18T00:07:02.942 INFO:teuthology.orchestra.run.smithi079.stdout: "average_load": 699, 2024-03-18T00:07:02.942 INFO:teuthology.orchestra.run.smithi079.stdout: "avg_session_uptime": 548, 2024-03-18T00:07:02.942 INFO:teuthology.orchestra.run.smithi079.stdout: "mdthresh_evicted": 0 2024-03-18T00:07:02.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.942 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool": { 2024-03-18T00:07:02.942 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_bytes": 0, 2024-03-18T00:07:02.942 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_items": 0, 2024-03-18T00:07:02.942 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T00:07:02.942 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_items": 0, 2024-03-18T00:07:02.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T00:07:02.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_items": 0, 2024-03-18T00:07:02.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T00:07:02.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T00:07:02.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T00:07:02.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T00:07:02.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T00:07:02.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_items": 0, 2024-03-18T00:07:02.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T00:07:02.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T00:07:02.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_bytes": 0, 2024-03-18T00:07:02.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_items": 0, 2024-03-18T00:07:02.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_bytes": 0, 2024-03-18T00:07:02.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_items": 0, 2024-03-18T00:07:02.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T00:07:02.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T00:07:02.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T00:07:02.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T00:07:02.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T00:07:02.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_items": 0, 2024-03-18T00:07:02.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_bytes": 0, 2024-03-18T00:07:02.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_items": 0, 2024-03-18T00:07:02.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T00:07:02.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T00:07:02.945 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_bytes": 0, 2024-03-18T00:07:02.945 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_items": 0, 2024-03-18T00:07:02.945 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_bytes": 0, 2024-03-18T00:07:02.945 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_items": 0, 2024-03-18T00:07:02.945 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T00:07:02.945 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_items": 0, 2024-03-18T00:07:02.945 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T00:07:02.945 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_items": 0, 2024-03-18T00:07:02.945 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_bytes": 293481, 2024-03-18T00:07:02.945 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_items": 131, 2024-03-18T00:07:02.945 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_bytes": 792, 2024-03-18T00:07:02.946 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_items": 9, 2024-03-18T00:07:02.946 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bytes": 0, 2024-03-18T00:07:02.946 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_items": 0, 2024-03-18T00:07:02.946 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_bytes": 0, 2024-03-18T00:07:02.946 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_items": 0, 2024-03-18T00:07:02.946 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_bytes": 0, 2024-03-18T00:07:02.946 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_items": 0, 2024-03-18T00:07:02.946 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_bytes": 8792, 2024-03-18T00:07:02.946 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_items": 168, 2024-03-18T00:07:02.946 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T00:07:02.946 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_items": 0, 2024-03-18T00:07:02.947 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_bytes": 0, 2024-03-18T00:07:02.947 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_items": 0, 2024-03-18T00:07:02.947 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_bytes": 141834, 2024-03-18T00:07:02.947 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_items": 846, 2024-03-18T00:07:02.947 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_bytes": 0, 2024-03-18T00:07:02.947 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_items": 0, 2024-03-18T00:07:02.947 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_bytes": 0, 2024-03-18T00:07:02.947 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_items": 0 2024-03-18T00:07:02.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.947 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter": { 2024-03-18T00:07:02.947 INFO:teuthology.orchestra.run.smithi079.stdout: "op_active": 0, 2024-03-18T00:07:02.948 INFO:teuthology.orchestra.run.smithi079.stdout: "op_laggy": 0, 2024-03-18T00:07:02.948 INFO:teuthology.orchestra.run.smithi079.stdout: "op_send": 3372, 2024-03-18T00:07:02.948 INFO:teuthology.orchestra.run.smithi079.stdout: "op_send_bytes": 9041297, 2024-03-18T00:07:02.948 INFO:teuthology.orchestra.run.smithi079.stdout: "op_resend": 37, 2024-03-18T00:07:02.948 INFO:teuthology.orchestra.run.smithi079.stdout: "op_reply": 3372, 2024-03-18T00:07:02.948 INFO:teuthology.orchestra.run.smithi079.stdout: "op_latency": { 2024-03-18T00:07:02.948 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 3372, 2024-03-18T00:07:02.948 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 63.574518703, 2024-03-18T00:07:02.948 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.018853653 2024-03-18T00:07:02.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.948 INFO:teuthology.orchestra.run.smithi079.stdout: "op_inflight": 0, 2024-03-18T00:07:02.948 INFO:teuthology.orchestra.run.smithi079.stdout: "oplen_avg": { 2024-03-18T00:07:02.949 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 3372, 2024-03-18T00:07:02.949 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 3600 2024-03-18T00:07:02.949 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.949 INFO:teuthology.orchestra.run.smithi079.stdout: "op": 3372, 2024-03-18T00:07:02.949 INFO:teuthology.orchestra.run.smithi079.stdout: "op_r": 75, 2024-03-18T00:07:02.949 INFO:teuthology.orchestra.run.smithi079.stdout: "op_w": 3297, 2024-03-18T00:07:02.949 INFO:teuthology.orchestra.run.smithi079.stdout: "op_rmw": 0, 2024-03-18T00:07:02.949 INFO:teuthology.orchestra.run.smithi079.stdout: "op_pg": 0, 2024-03-18T00:07:02.949 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_stat": 0, 2024-03-18T00:07:02.949 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_create": 11, 2024-03-18T00:07:02.949 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_read": 75, 2024-03-18T00:07:02.950 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_write": 493, 2024-03-18T00:07:02.950 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_writefull": 91, 2024-03-18T00:07:02.950 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_writesame": 0, 2024-03-18T00:07:02.950 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_append": 0, 2024-03-18T00:07:02.950 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_zero": 0, 2024-03-18T00:07:02.950 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_truncate": 2, 2024-03-18T00:07:02.950 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_delete": 2507, 2024-03-18T00:07:02.950 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_mapext": 0, 2024-03-18T00:07:02.950 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_sparse_read": 0, 2024-03-18T00:07:02.950 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_clonerange": 0, 2024-03-18T00:07:02.950 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_getxattr": 0, 2024-03-18T00:07:02.951 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_setxattr": 22, 2024-03-18T00:07:02.951 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_cmpxattr": 0, 2024-03-18T00:07:02.951 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_rmxattr": 0, 2024-03-18T00:07:02.951 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_resetxattrs": 0, 2024-03-18T00:07:02.951 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_call": 0, 2024-03-18T00:07:02.951 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_watch": 0, 2024-03-18T00:07:02.951 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_notify": 0, 2024-03-18T00:07:02.951 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T00:07:02.951 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_pgls": 0, 2024-03-18T00:07:02.951 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_pgls_filter": 0, 2024-03-18T00:07:02.951 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_other": 5, 2024-03-18T00:07:02.951 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_active": 0, 2024-03-18T00:07:02.952 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_send": 0, 2024-03-18T00:07:02.952 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_resend": 0, 2024-03-18T00:07:02.952 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_ping": 0, 2024-03-18T00:07:02.952 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_active": 0, 2024-03-18T00:07:02.952 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_send": 0, 2024-03-18T00:07:02.952 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_resend": 0, 2024-03-18T00:07:02.952 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_active": 0, 2024-03-18T00:07:02.952 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_send": 0, 2024-03-18T00:07:02.952 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_resend": 0, 2024-03-18T00:07:02.952 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_active": 0, 2024-03-18T00:07:02.952 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_send": 0, 2024-03-18T00:07:02.953 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_resend": 0, 2024-03-18T00:07:02.953 INFO:teuthology.orchestra.run.smithi079.stdout: "command_active": 0, 2024-03-18T00:07:02.953 INFO:teuthology.orchestra.run.smithi079.stdout: "command_send": 0, 2024-03-18T00:07:02.953 INFO:teuthology.orchestra.run.smithi079.stdout: "command_resend": 0, 2024-03-18T00:07:02.953 INFO:teuthology.orchestra.run.smithi079.stdout: "map_epoch": 94, 2024-03-18T00:07:02.953 INFO:teuthology.orchestra.run.smithi079.stdout: "map_full": 0, 2024-03-18T00:07:02.953 INFO:teuthology.orchestra.run.smithi079.stdout: "map_inc": 10, 2024-03-18T00:07:02.953 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_sessions": 12, 2024-03-18T00:07:02.953 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_session_open": 12, 2024-03-18T00:07:02.953 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_session_close": 0, 2024-03-18T00:07:02.953 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_laggy": 0, 2024-03-18T00:07:02.953 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_wr": 324, 2024-03-18T00:07:02.954 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_rd": 0, 2024-03-18T00:07:02.954 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_del": 70 2024-03-18T00:07:02.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.954 INFO:teuthology.orchestra.run.smithi079.stdout: "oft": { 2024-03-18T00:07:02.954 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_objs": 1, 2024-03-18T00:07:02.954 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_kv_pairs": 0, 2024-03-18T00:07:02.954 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_updates": 0, 2024-03-18T00:07:02.954 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_removes": 0 2024-03-18T00:07:02.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.954 INFO:teuthology.orchestra.run.smithi079.stdout: "osd-slow-ops": { 2024-03-18T00:07:02.954 INFO:teuthology.orchestra.run.smithi079.stdout: "slow_ops_count": 0 2024-03-18T00:07:02.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.955 INFO:teuthology.orchestra.run.smithi079.stdout: "purge_queue": { 2024-03-18T00:07:02.955 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_ops": 0, 2024-03-18T00:07:02.955 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_ops_high_water": 10, 2024-03-18T00:07:02.955 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing": 0, 2024-03-18T00:07:02.955 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_high_water": 5, 2024-03-18T00:07:02.955 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executed_ops": 1993, 2024-03-18T00:07:02.955 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executed": 1003, 2024-03-18T00:07:02.955 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_item_in_journal": 0 2024-03-18T00:07:02.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.955 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T00:07:02.955 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:02.955 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2024-03-18T00:07:02.956 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:02.956 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 40517, 2024-03-18T00:07:02.956 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 12438300, 2024-03-18T00:07:02.956 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:02.956 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 40517, 2024-03-18T00:07:02.956 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-03-18T00:07:02.956 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-03-18T00:07:02.956 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 40517, 2024-03-18T00:07:02.956 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 12438300, 2024-03-18T00:07:02.956 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:02.956 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.956 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.956 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.957 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:02.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.957 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-objecter_bytes": { 2024-03-18T00:07:02.957 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:02.957 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2024-03-18T00:07:02.957 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:02.957 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2024-03-18T00:07:02.957 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2024-03-18T00:07:02.957 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:02.957 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:02.957 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 3372, 2024-03-18T00:07:02.958 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 9150520, 2024-03-18T00:07:02.958 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 3372, 2024-03-18T00:07:02.958 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 9150520, 2024-03-18T00:07:02.958 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:02.958 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.958 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.958 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.958 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:02.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.958 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-objecter_ops": { 2024-03-18T00:07:02.958 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:02.958 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 1024, 2024-03-18T00:07:02.959 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:02.959 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2024-03-18T00:07:02.959 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2024-03-18T00:07:02.959 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:02.959 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:02.959 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 3372, 2024-03-18T00:07:02.959 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 3372, 2024-03-18T00:07:02.959 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 3372, 2024-03-18T00:07:02.959 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 3372, 2024-03-18T00:07:02.959 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:02.959 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.959 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.959 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.960 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:02.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.960 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-write_buf_throttle": { 2024-03-18T00:07:02.960 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:02.960 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 3758096384, 2024-03-18T00:07:02.960 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:02.960 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 1003, 2024-03-18T00:07:02.960 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 109223, 2024-03-18T00:07:02.960 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:02.960 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 1003, 2024-03-18T00:07:02.960 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-03-18T00:07:02.960 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-03-18T00:07:02.961 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 75, 2024-03-18T00:07:02.961 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 109223, 2024-03-18T00:07:02.961 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:02.961 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.961 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.961 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.961 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:02.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:02.961 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-write_buf_throttle-0x55de08a32500": { 2024-03-18T00:07:02.961 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:02.961 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 3758096384, 2024-03-18T00:07:02.961 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:02.961 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 4872, 2024-03-18T00:07:02.962 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 8448834, 2024-03-18T00:07:02.962 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:02.962 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 4872, 2024-03-18T00:07:02.962 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-03-18T00:07:02.962 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-03-18T00:07:02.962 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 418, 2024-03-18T00:07:02.962 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 8448834, 2024-03-18T00:07:02.962 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:02.962 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:02.962 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:02.962 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:02.962 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:02.963 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:02.963 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-03-18T00:07:03.642 INFO:tasks.check_counter:Daemon mds.b mds.exported=17 2024-03-18T00:07:03.642 INFO:tasks.check_counter:Daemon mds.b mds.imported=3 2024-03-18T00:07:03.642 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=353 2024-03-18T00:07:03.642 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=4859 2024-03-18T00:07:03.642 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=2 2024-03-18T00:07:03.642 DEBUG:tasks.check_counter:Getting stats from e 2024-03-18T00:07:03.642 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-18T00:07:03.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:03 smithi012 ceph-mon[31541]: pgmap v688: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:07:03.972 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T00:07:04.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:03 smithi079 ceph-mon[36241]: pgmap v688: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:07:04.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:03 smithi191 ceph-mon[36708]: pgmap v688: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:07:05.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:05 smithi012 ceph-mon[31541]: pgmap v689: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-18T00:07:05.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:05 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:07:05.972 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:05 smithi079 ceph-mon[36241]: pgmap v689: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-18T00:07:05.972 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:05 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:07:06.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:05 smithi191 ceph-mon[36708]: pgmap v689: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-18T00:07:06.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:05 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:07:06.402 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-03-18T00:07:06.403 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T00:07:06.403 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 1332, 2024-03-18T00:07:06.403 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 1305, 2024-03-18T00:07:06.403 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 5437224, 2024-03-18T00:07:06.403 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 473794, 2024-03-18T00:07:06.403 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 5, 2024-03-18T00:07:06.403 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 1, 2024-03-18T00:07:06.403 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.108223730, 2024-03-18T00:07:06.403 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.036320416, 2024-03-18T00:07:06.403 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.041313572, 2024-03-18T00:07:06.403 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.028905472, 2024-03-18T00:07:06.403 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 1305, 2024-03-18T00:07:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.037169010, 2024-03-18T00:07:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000028482 2024-03-18T00:07:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 905040, 2024-03-18T00:07:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 127488 2024-03-18T00:07:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T00:07:06.405 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 409, 2024-03-18T00:07:06.405 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 563, 2024-03-18T00:07:06.405 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 161858, 2024-03-18T00:07:06.405 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 468754, 2024-03-18T00:07:06.405 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 5, 2024-03-18T00:07:06.405 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 4, 2024-03-18T00:07:06.405 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.034420279, 2024-03-18T00:07:06.405 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.012088713, 2024-03-18T00:07:06.405 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.007267726, 2024-03-18T00:07:06.405 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.011901721, 2024-03-18T00:07:06.405 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:06.405 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 561, 2024-03-18T00:07:06.405 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.020462986, 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000036475 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 338944 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 770, 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 772, 2024-03-18T00:07:06.406 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 4389654, 2024-03-18T00:07:06.407 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 244867, 2024-03-18T00:07:06.407 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 4, 2024-03-18T00:07:06.407 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 3, 2024-03-18T00:07:06.407 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.053212813, 2024-03-18T00:07:06.407 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.014052533, 2024-03-18T00:07:06.407 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.016470506, 2024-03-18T00:07:06.407 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.018306121, 2024-03-18T00:07:06.407 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:06.407 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 770, 2024-03-18T00:07:06.407 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.022569029, 2024-03-18T00:07:06.407 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000029310 2024-03-18T00:07:06.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.407 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: "cct": { 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: "total_workers": 1, 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: "unhealthy_workers": 0 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: "finisher-MDSRank": { 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: "queue_len": 0, 2024-03-18T00:07:06.408 INFO:teuthology.orchestra.run.smithi079.stdout: "complete_latency": { 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 3776, 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.082683443, 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000021897 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: "finisher-PurgeQueue": { 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: "queue_len": 0, 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: "complete_latency": { 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.409 INFO:teuthology.orchestra.run.smithi079.stdout: "mds": { 2024-03-18T00:07:06.410 INFO:teuthology.orchestra.run.smithi079.stdout: "request": 0, 2024-03-18T00:07:06.410 INFO:teuthology.orchestra.run.smithi079.stdout: "reply": 0, 2024-03-18T00:07:06.410 INFO:teuthology.orchestra.run.smithi079.stdout: "reply_latency": { 2024-03-18T00:07:06.410 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.410 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.410 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.410 INFO:teuthology.orchestra.run.smithi079.stdout: "slow_reply": 0, 2024-03-18T00:07:06.410 INFO:teuthology.orchestra.run.smithi079.stdout: "forward": 0, 2024-03-18T00:07:06.410 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_fetch_complete": 0, 2024-03-18T00:07:06.410 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_fetch_keys": 0, 2024-03-18T00:07:06.410 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_commit": 0, 2024-03-18T00:07:06.410 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_split": 0, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_merge": 0, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes": 1022, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_top": 0, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_bottom": 1022, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_pin_tail": 0, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_pinned": 1022, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_expired": 0, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_with_caps": 0, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": 0, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "subtrees": 2, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse": 0, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_hit": 0, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_forward": 0, 2024-03-18T00:07:06.411 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_discover": 0, 2024-03-18T00:07:06.412 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_dir_fetch": 0, 2024-03-18T00:07:06.412 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_remote_ino": 0, 2024-03-18T00:07:06.412 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_lock": 0, 2024-03-18T00:07:06.412 INFO:teuthology.orchestra.run.smithi079.stdout: "load_cent": 0, 2024-03-18T00:07:06.412 INFO:teuthology.orchestra.run.smithi079.stdout: "q": 0, 2024-03-18T00:07:06.412 INFO:teuthology.orchestra.run.smithi079.stdout: "exported": 0, 2024-03-18T00:07:06.412 INFO:teuthology.orchestra.run.smithi079.stdout: "exported_inodes": 0, 2024-03-18T00:07:06.412 INFO:teuthology.orchestra.run.smithi079.stdout: "imported": 0, 2024-03-18T00:07:06.412 INFO:teuthology.orchestra.run.smithi079.stdout: "imported_inodes": 0, 2024-03-18T00:07:06.412 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_dir_fetch": 0, 2024-03-18T00:07:06.412 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_backtrace_fetch": 0, 2024-03-18T00:07:06.412 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_peer_discover": 0, 2024-03-18T00:07:06.412 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rfiles": 4, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rbytes": 234, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rsnaps": 0, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_set_tag": 0, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_inotable_repaired": 0, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dir_inodes": 0, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_file_inodes": 0, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_inode_file_caps": 0, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_grant": 0, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T00:07:06.413 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_caps": 0, 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_caps_dirty": 0, 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_cap_release": 0, 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: "process_request_cap_release": 0 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache": { 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_update": 0, 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_update_receipt": 0, 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_try_discover": 0, 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_send_discover": 0, 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_handle_discover": 0, 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays": 0, 2024-03-18T00:07:06.414 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays_delayed": 0, 2024-03-18T00:07:06.415 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays_enqueuing": 0, 2024-03-18T00:07:06.415 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_created": 0, 2024-03-18T00:07:06.415 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_enqueued": 0, 2024-03-18T00:07:06.415 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_reintegrated": 0, 2024-03-18T00:07:06.415 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_migrated": 0, 2024-03-18T00:07:06.415 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_processing": 0, 2024-03-18T00:07:06.415 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_enqueued": 0, 2024-03-18T00:07:06.415 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_prioritized": 0, 2024-03-18T00:07:06.415 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_started": 0, 2024-03-18T00:07:06.415 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_completed": 0, 2024-03-18T00:07:06.415 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_quiesce_path": 0, 2024-03-18T00:07:06.416 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_quiesce_inode": 0, 2024-03-18T00:07:06.416 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T00:07:06.416 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_exportdir": 0, 2024-03-18T00:07:06.416 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_flush": 0, 2024-03-18T00:07:06.416 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_fragmentdir": 0, 2024-03-18T00:07:06.416 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_fragstats": 0, 2024-03-18T00:07:06.416 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_inodestats": 0 2024-03-18T00:07:06.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.416 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log": { 2024-03-18T00:07:06.416 INFO:teuthology.orchestra.run.smithi079.stdout: "evlrg": 0, 2024-03-18T00:07:06.416 INFO:teuthology.orchestra.run.smithi079.stdout: "evadd": 0, 2024-03-18T00:07:06.416 INFO:teuthology.orchestra.run.smithi079.stdout: "evex": 0, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "evtrm": 0, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "ev": 4872, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "evexg": 0, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "evexd": 0, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "segadd": 0, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "segex": 0, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "segtrm": 0, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "seg": 8, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "segmjr": 2, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "segexg": 0, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "segexd": 0, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "expos": 4194304, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "wrpos": 12643138, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "rdpos": 12642539, 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "jlat": { 2024-03-18T00:07:06.417 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.418 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.418 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.418 INFO:teuthology.orchestra.run.smithi079.stdout: "replayed": 4872 2024-03-18T00:07:06.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.418 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_mem": { 2024-03-18T00:07:06.418 INFO:teuthology.orchestra.run.smithi079.stdout: "ino": 19, 2024-03-18T00:07:06.418 INFO:teuthology.orchestra.run.smithi079.stdout: "ino+": 1048, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "ino-": 1029, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "dir": 14, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "dir+": 56, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "dir-": 42, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "dn": 1022, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "dn+": 2012, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "dn-": 990, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "cap": 0, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "cap+": 0, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "cap-": 0, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "rss": 50452, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "heap": 215420 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_server": { 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "dispatch_client_request": 0, 2024-03-18T00:07:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: "dispatch_server_request": 0, 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_request": 0, 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_session": 0, 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_peer_request": 0, 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: "req_create_latency": { 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getattr_latency": { 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getfilelock_latency": { 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: "req_link_latency": { 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookup_latency": { 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookuphash_latency": { 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupino_latency": { 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupname_latency": { 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupparent_latency": { 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupsnap_latency": { 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lssnap_latency": { 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.423 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.424 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.424 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mkdir_latency": { 2024-03-18T00:07:06.424 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.424 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.424 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.425 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mknod_latency": { 2024-03-18T00:07:06.425 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.425 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.425 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.425 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mksnap_latency": { 2024-03-18T00:07:06.425 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.425 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.425 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.425 INFO:teuthology.orchestra.run.smithi079.stdout: "req_open_latency": { 2024-03-18T00:07:06.425 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: "req_readdir_latency": { 2024-03-18T00:07:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rename_latency": { 2024-03-18T00:07:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: "req_renamesnap_latency": { 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: "req_snapdiff_latency": { 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmdir_latency": { 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.428 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmsnap_latency": { 2024-03-18T00:07:06.428 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.428 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.428 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.428 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmxattr_latency": { 2024-03-18T00:07:06.428 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.428 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.428 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.428 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setattr_latency": { 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setdirlayout_latency": { 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setfilelock_latency": { 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.429 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setlayout_latency": { 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setxattr_latency": { 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: "req_symlink_latency": { 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.431 INFO:teuthology.orchestra.run.smithi079.stdout: "req_unlink_latency": { 2024-03-18T00:07:06.431 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.431 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.431 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.431 INFO:teuthology.orchestra.run.smithi079.stdout: "cap_revoke_eviction": 0, 2024-03-18T00:07:06.431 INFO:teuthology.orchestra.run.smithi079.stdout: "cap_acquisition_throttle": 0, 2024-03-18T00:07:06.431 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getvxattr_latency": { 2024-03-18T00:07:06.431 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.431 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.431 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.431 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:06.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_sessions": { 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: "session_count": 1, 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: "session_add": 4, 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: "session_remove": 3, 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: "sessions_open": 2, 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: "sessions_stale": 0, 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: "total_load": 0, 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: "average_load": 0, 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: "avg_session_uptime": 4607, 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: "mdthresh_evicted": 0 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool": { 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_bytes": 0, 2024-03-18T00:07:06.432 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_items": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_items": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_items": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_items": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_bytes": 0, 2024-03-18T00:07:06.433 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_items": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_bytes": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_items": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_items": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_bytes": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_items": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_bytes": 0, 2024-03-18T00:07:06.434 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_items": 0, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_bytes": 0, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_items": 0, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_items": 0, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_items": 0, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_bytes": 189401, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_items": 51, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_bytes": 0, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_items": 0, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bytes": 0, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_items": 0, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_bytes": 0, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_items": 0, 2024-03-18T00:07:06.435 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_bytes": 0, 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_items": 0, 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_bytes": 8792, 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_items": 168, 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_items": 0, 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_bytes": 0, 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_items": 0, 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_bytes": 779722, 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_items": 4337, 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_bytes": 0, 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_items": 0, 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_bytes": 0, 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_items": 0 2024-03-18T00:07:06.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.437 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter": { 2024-03-18T00:07:06.437 INFO:teuthology.orchestra.run.smithi079.stdout: "op_active": 0, 2024-03-18T00:07:06.437 INFO:teuthology.orchestra.run.smithi079.stdout: "op_laggy": 0, 2024-03-18T00:07:06.437 INFO:teuthology.orchestra.run.smithi079.stdout: "op_send": 2267, 2024-03-18T00:07:06.437 INFO:teuthology.orchestra.run.smithi079.stdout: "op_send_bytes": 26, 2024-03-18T00:07:06.437 INFO:teuthology.orchestra.run.smithi079.stdout: "op_resend": 4, 2024-03-18T00:07:06.437 INFO:teuthology.orchestra.run.smithi079.stdout: "op_reply": 2267, 2024-03-18T00:07:06.437 INFO:teuthology.orchestra.run.smithi079.stdout: "op_latency": { 2024-03-18T00:07:06.437 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 2267, 2024-03-18T00:07:06.437 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 2.273947505, 2024-03-18T00:07:06.437 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.001003064 2024-03-18T00:07:06.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.437 INFO:teuthology.orchestra.run.smithi079.stdout: "op_inflight": 0, 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: "oplen_avg": { 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 2267, 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 2268 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: "op": 2267, 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: "op_r": 2267, 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: "op_w": 0, 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: "op_rmw": 0, 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: "op_pg": 0, 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_stat": 1392, 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_create": 0, 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_read": 872, 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_write": 0, 2024-03-18T00:07:06.438 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_writefull": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_writesame": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_append": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_zero": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_truncate": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_delete": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_mapext": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_sparse_read": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_clonerange": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_getxattr": 2, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_setxattr": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_cmpxattr": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_rmxattr": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_resetxattrs": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_call": 0, 2024-03-18T00:07:06.439 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_watch": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_notify": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_pgls": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_pgls_filter": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_other": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_active": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_send": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_resend": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_ping": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_active": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_send": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_resend": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_active": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_send": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_resend": 0, 2024-03-18T00:07:06.440 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_active": 0, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_send": 0, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_resend": 0, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "command_active": 0, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "command_send": 0, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "command_resend": 0, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "map_epoch": 94, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "map_full": 0, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "map_inc": 4, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_sessions": 5, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_session_open": 5, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_session_close": 0, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_laggy": 0, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_wr": 0, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_rd": 2, 2024-03-18T00:07:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_del": 0 2024-03-18T00:07:06.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.442 INFO:teuthology.orchestra.run.smithi079.stdout: "oft": { 2024-03-18T00:07:06.442 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_objs": 0, 2024-03-18T00:07:06.442 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_kv_pairs": 0, 2024-03-18T00:07:06.442 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_updates": 0, 2024-03-18T00:07:06.442 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_removes": 0 2024-03-18T00:07:06.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.442 INFO:teuthology.orchestra.run.smithi079.stdout: "osd-slow-ops": { 2024-03-18T00:07:06.442 INFO:teuthology.orchestra.run.smithi079.stdout: "slow_ops_count": 0 2024-03-18T00:07:06.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.442 INFO:teuthology.orchestra.run.smithi079.stdout: "purge_queue": { 2024-03-18T00:07:06.442 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_ops": 0, 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing": 0, 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_high_water": 0, 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executed_ops": 0, 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executed": 0, 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_item_in_journal": 0 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 2507, 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 9747880, 2024-03-18T00:07:06.443 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:06.444 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 2507, 2024-03-18T00:07:06.444 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-03-18T00:07:06.444 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-03-18T00:07:06.444 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 2507, 2024-03-18T00:07:06.444 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 9747880, 2024-03-18T00:07:06.444 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:06.444 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.444 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.444 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.444 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:06.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-objecter_bytes": { 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 2267, 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 8450452, 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 2267, 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 8450452, 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:06.445 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.446 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.446 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.446 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:06.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.446 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-objecter_ops": { 2024-03-18T00:07:06.446 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:06.446 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 1024, 2024-03-18T00:07:06.446 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:06.446 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2024-03-18T00:07:06.446 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2024-03-18T00:07:06.446 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:06.446 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:06.446 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 2267, 2024-03-18T00:07:06.447 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 2267, 2024-03-18T00:07:06.447 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 2267, 2024-03-18T00:07:06.447 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 2267, 2024-03-18T00:07:06.447 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:06.447 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.447 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.447 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.447 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:06.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.447 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-write_buf_throttle": { 2024-03-18T00:07:06.447 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:06.447 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 3758096384, 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 0, 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 0, 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.448 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-write_buf_throttle-0x55f5312466e0": { 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 3758096384, 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 0, 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 0, 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:06.449 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:06.450 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:06.450 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:06.450 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:06.450 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:06.450 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-03-18T00:07:07.161 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-18T00:07:07.161 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-18T00:07:07.161 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-18T00:07:07.161 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-18T00:07:07.161 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-18T00:07:07.161 DEBUG:tasks.check_counter:Getting stats from h 2024-03-18T00:07:07.162 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-18T00:07:07.474 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T00:07:07.902 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:07 smithi079 ceph-mon[36241]: pgmap v690: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-18T00:07:07.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:07 smithi012 ceph-mon[31541]: pgmap v690: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-18T00:07:08.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:07 smithi191 ceph-mon[36708]: pgmap v690: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-18T00:07:09.305 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-03-18T00:07:09.305 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T00:07:09.305 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 340589, 2024-03-18T00:07:09.305 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 554399, 2024-03-18T00:07:09.305 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 116261951, 2024-03-18T00:07:09.305 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 457329442, 2024-03-18T00:07:09.305 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 15, 2024-03-18T00:07:09.305 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 6, 2024-03-18T00:07:09.306 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 23.764798318, 2024-03-18T00:07:09.306 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 10.207375753, 2024-03-18T00:07:09.306 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 31.035081463, 2024-03-18T00:07:09.306 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 2.875607545, 2024-03-18T00:07:09.306 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:09.306 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 554397, 2024-03-18T00:07:09.306 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 68.982801090, 2024-03-18T00:07:09.306 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000124428 2024-03-18T00:07:09.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.306 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:09.306 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 651078, 2024-03-18T00:07:09.306 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.121340332, 2024-03-18T00:07:09.306 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000186 2024-03-18T00:07:09.307 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.307 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T00:07:09.307 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T00:07:09.307 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.307 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T00:07:09.307 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 71743, 2024-03-18T00:07:09.307 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 73134, 2024-03-18T00:07:09.307 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 19823574, 2024-03-18T00:07:09.307 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 341593695, 2024-03-18T00:07:09.307 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 9, 2024-03-18T00:07:09.307 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 8, 2024-03-18T00:07:09.307 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 7.913430218, 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 3.147988458, 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 1.746923538, 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 2.762155521, 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 73134, 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 45.709513386, 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000625010 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 28285, 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.005159299, 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000182 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 931504, 2024-03-18T00:07:09.308 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 137168 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 57963, 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 63325, 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 18784155, 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 160992818, 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 8, 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 6, 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 7.319583612, 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 3.763581458, 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 1.819307408, 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 1.734209858, 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:09.309 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 63323, 2024-03-18T00:07:09.310 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 23.860781009, 2024-03-18T00:07:09.310 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000376810 2024-03-18T00:07:09.310 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.310 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:09.310 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 57360, 2024-03-18T00:07:09.310 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.011295319, 2024-03-18T00:07:09.310 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000196 2024-03-18T00:07:09.310 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.310 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T00:07:09.310 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 342000 2024-03-18T00:07:09.310 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.310 INFO:teuthology.orchestra.run.smithi079.stdout: "cct": { 2024-03-18T00:07:09.310 INFO:teuthology.orchestra.run.smithi079.stdout: "total_workers": 1, 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: "unhealthy_workers": 0 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: "finisher-MDSRank": { 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: "queue_len": 0, 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: "complete_latency": { 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 3461, 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 306.238786959, 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.088482746 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: "finisher-PurgeQueue": { 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: "queue_len": 0, 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: "complete_latency": { 2024-03-18T00:07:09.311 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 92497, 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 40.241608102, 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000435058 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mds": { 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: "request": 160205, 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: "reply": 158016, 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: "reply_latency": { 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 158016, 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 3599.109279833, 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.022776866 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: "slow_reply": 0, 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: "forward": 2189, 2024-03-18T00:07:09.312 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_fetch_complete": 1, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_fetch_keys": 0, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_commit": 2342, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_split": 155, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_merge": 7, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes": 148, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_top": 93, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_bottom": 43, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_pin_tail": 12, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_pinned": 14, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_expired": 3348, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_with_caps": 0, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": 0, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "subtrees": 6, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse": 229718, 2024-03-18T00:07:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_hit": 217582, 2024-03-18T00:07:09.314 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_forward": 2171, 2024-03-18T00:07:09.314 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_discover": 98, 2024-03-18T00:07:09.314 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_dir_fetch": 0, 2024-03-18T00:07:09.314 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_remote_ino": 0, 2024-03-18T00:07:09.314 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_lock": 157, 2024-03-18T00:07:09.314 INFO:teuthology.orchestra.run.smithi079.stdout: "load_cent": 0, 2024-03-18T00:07:09.314 INFO:teuthology.orchestra.run.smithi079.stdout: "q": 0, 2024-03-18T00:07:09.314 INFO:teuthology.orchestra.run.smithi079.stdout: "exported": 14, 2024-03-18T00:07:09.314 INFO:teuthology.orchestra.run.smithi079.stdout: "exported_inodes": 1958, 2024-03-18T00:07:09.314 INFO:teuthology.orchestra.run.smithi079.stdout: "imported": 98, 2024-03-18T00:07:09.314 INFO:teuthology.orchestra.run.smithi079.stdout: "imported_inodes": 290, 2024-03-18T00:07:09.314 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_dir_fetch": 0, 2024-03-18T00:07:09.314 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_backtrace_fetch": 0, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_peer_discover": 0, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rfiles": 2278, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rbytes": 28871204, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rsnaps": 0, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_set_tag": 0, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_inotable_repaired": 0, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dir_inodes": 0, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_file_inodes": 0, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_inode_file_caps": 26, 2024-03-18T00:07:09.315 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_revoke": 43320, 2024-03-18T00:07:09.316 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_grant": 103061, 2024-03-18T00:07:09.316 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_trunc": 2, 2024-03-18T00:07:09.316 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T00:07:09.316 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T00:07:09.316 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_caps": 112757, 2024-03-18T00:07:09.316 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_caps_dirty": 50279, 2024-03-18T00:07:09.316 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_cap_release": 3511, 2024-03-18T00:07:09.316 INFO:teuthology.orchestra.run.smithi079.stdout: "process_request_cap_release": 100363 2024-03-18T00:07:09.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache": { 2024-03-18T00:07:09.316 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_update": 4910, 2024-03-18T00:07:09.316 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_update_receipt": 15, 2024-03-18T00:07:09.316 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_try_discover": 10, 2024-03-18T00:07:09.317 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_send_discover": 87, 2024-03-18T00:07:09.317 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_handle_discover": 11766, 2024-03-18T00:07:09.317 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays": 2, 2024-03-18T00:07:09.317 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays_delayed": 0, 2024-03-18T00:07:09.317 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays_enqueuing": 0, 2024-03-18T00:07:09.317 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_created": 48523, 2024-03-18T00:07:09.317 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_enqueued": 48521, 2024-03-18T00:07:09.317 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_reintegrated": 0, 2024-03-18T00:07:09.317 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_migrated": 0, 2024-03-18T00:07:09.317 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_processing": 0, 2024-03-18T00:07:09.317 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_enqueued": 0, 2024-03-18T00:07:09.317 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_prioritized": 0, 2024-03-18T00:07:09.317 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_started": 0, 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_completed": 0, 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_quiesce_path": 0, 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_quiesce_inode": 0, 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_exportdir": 14, 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_flush": 0, 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_fragmentdir": 166, 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_fragstats": 0, 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_inodestats": 0 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log": { 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: "evlrg": 0, 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: "evadd": 265375, 2024-03-18T00:07:09.318 INFO:teuthology.orchestra.run.smithi079.stdout: "evex": 165125, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "evtrm": 159771, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "ev": 105604, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "evexg": 0, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "evexd": 5354, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "segadd": 336, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "segex": 208, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "segtrm": 201, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "seg": 135, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "segmjr": 9, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "segexg": 0, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "segexd": 7, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "expos": 392639585, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "wrpos": 597083716, 2024-03-18T00:07:09.319 INFO:teuthology.orchestra.run.smithi079.stdout: "rdpos": 0, 2024-03-18T00:07:09.320 INFO:teuthology.orchestra.run.smithi079.stdout: "jlat": { 2024-03-18T00:07:09.320 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 2282, 2024-03-18T00:07:09.320 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 677.582431193, 2024-03-18T00:07:09.320 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.296924816 2024-03-18T00:07:09.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.320 INFO:teuthology.orchestra.run.smithi079.stdout: "replayed": 0 2024-03-18T00:07:09.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.320 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_mem": { 2024-03-18T00:07:09.320 INFO:teuthology.orchestra.run.smithi079.stdout: "ino": 43, 2024-03-18T00:07:09.320 INFO:teuthology.orchestra.run.smithi079.stdout: "ino+": 49563, 2024-03-18T00:07:09.320 INFO:teuthology.orchestra.run.smithi079.stdout: "ino-": 49520, 2024-03-18T00:07:09.320 INFO:teuthology.orchestra.run.smithi079.stdout: "dir": 52, 2024-03-18T00:07:09.320 INFO:teuthology.orchestra.run.smithi079.stdout: "dir+": 1588, 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: "dir-": 1536, 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: "dn": 148, 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: "dn+": 113243, 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: "dn-": 113095, 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: "cap": 0, 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: "cap+": 49614, 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: "cap-": 49614, 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: "rss": 153364, 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: "heap": 223612 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_server": { 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: "dispatch_client_request": 168331, 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: "dispatch_server_request": 7866, 2024-03-18T00:07:09.321 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_request": 160205, 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_session": 25, 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_peer_request": 16212, 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: "req_create_latency": { 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 49209, 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 853.791177583, 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.017350305 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getattr_latency": { 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 2838, 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 956.291569485, 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.336959679 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getfilelock_latency": { 2024-03-18T00:07:09.322 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: "req_link_latency": { 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookup_latency": { 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 16186, 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 352.891389634, 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.021802260 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookuphash_latency": { 2024-03-18T00:07:09.323 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupino_latency": { 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupname_latency": { 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupparent_latency": { 2024-03-18T00:07:09.324 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.325 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.325 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.325 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupsnap_latency": { 2024-03-18T00:07:09.325 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.325 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.326 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.326 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lssnap_latency": { 2024-03-18T00:07:09.326 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.326 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.326 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.326 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mkdir_latency": { 2024-03-18T00:07:09.326 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 190, 2024-03-18T00:07:09.326 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 7.411179898, 2024-03-18T00:07:09.326 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.039006209 2024-03-18T00:07:09.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mknod_latency": { 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mksnap_latency": { 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: "req_open_latency": { 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 9718, 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 266.418373487, 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.027414938 2024-03-18T00:07:09.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: "req_readdir_latency": { 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 1245, 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 3.253120750, 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.002612948 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rename_latency": { 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 49183, 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 1055.149048990, 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.021453531 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: "req_renamesnap_latency": { 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.328 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: "req_snapdiff_latency": { 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmdir_latency": { 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 190, 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 33.608527785, 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.176886988 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmsnap_latency": { 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmxattr_latency": { 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setattr_latency": { 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 2, 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 1.162952241, 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.581476120 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setdirlayout_latency": { 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.330 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setfilelock_latency": { 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setlayout_latency": { 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setxattr_latency": { 2024-03-18T00:07:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: "req_symlink_latency": { 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: "req_unlink_latency": { 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 29255, 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 69.131939980, 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.002363081 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.332 INFO:teuthology.orchestra.run.smithi079.stdout: "cap_revoke_eviction": 0, 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: "cap_acquisition_throttle": 0, 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getvxattr_latency": { 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_sessions": { 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: "session_count": 1, 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: "session_add": 2, 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: "session_remove": 1, 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: "sessions_open": 1, 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: "sessions_stale": 0, 2024-03-18T00:07:09.333 INFO:teuthology.orchestra.run.smithi079.stdout: "total_load": 27315, 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: "average_load": 27315, 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: "avg_session_uptime": 554, 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: "mdthresh_evicted": 0 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool": { 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_bytes": 0, 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_items": 0, 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_items": 0, 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_items": 0, 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T00:07:09.334 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_items": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_bytes": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_items": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_bytes": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_items": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T00:07:09.335 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_items": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_bytes": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_items": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_bytes": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_items": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_bytes": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_items": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_items": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_items": 0, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_bytes": 321945, 2024-03-18T00:07:09.336 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_items": 126, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_bytes": 0, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_items": 0, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bytes": 0, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_items": 0, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_bytes": 0, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_items": 0, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_bytes": 0, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_items": 0, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_bytes": 8792, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_items": 168, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_items": 0, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_bytes": 0, 2024-03-18T00:07:09.337 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_items": 0, 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_bytes": 274981, 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_items": 1655, 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_bytes": 0, 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_items": 0, 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_bytes": 0, 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_items": 0 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter": { 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: "op_active": 0, 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: "op_laggy": 0, 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: "op_send": 142184, 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: "op_send_bytes": 661250489, 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: "op_resend": 33, 2024-03-18T00:07:09.338 INFO:teuthology.orchestra.run.smithi079.stdout: "op_reply": 142184, 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: "op_latency": { 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 142184, 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 9651.614314551, 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.067881156 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: "op_inflight": 0, 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: "oplen_avg": { 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 142184, 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 201040 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: "op": 142184, 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: "op_r": 1127, 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: "op_w": 141057, 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: "op_rmw": 0, 2024-03-18T00:07:09.339 INFO:teuthology.orchestra.run.smithi079.stdout: "op_pg": 0, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_stat": 924, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_create": 35735, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_read": 1126, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_write": 3408, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_writefull": 476, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_writesame": 0, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_append": 0, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_zero": 0, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_truncate": 155, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_delete": 98429, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_mapext": 0, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_sparse_read": 0, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_clonerange": 0, 2024-03-18T00:07:09.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_getxattr": 1, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_setxattr": 53682, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_cmpxattr": 0, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_rmxattr": 0, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_resetxattrs": 0, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_call": 0, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_watch": 0, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_notify": 0, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_pgls": 0, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_pgls_filter": 0, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_other": 2, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_active": 0, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_send": 0, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_resend": 0, 2024-03-18T00:07:09.341 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_ping": 0, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_active": 0, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_send": 0, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_resend": 0, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_active": 0, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_send": 0, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_resend": 0, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_active": 0, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_send": 0, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_resend": 0, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "command_active": 0, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "command_send": 0, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "command_resend": 0, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "map_epoch": 94, 2024-03-18T00:07:09.342 INFO:teuthology.orchestra.run.smithi079.stdout: "map_full": 0, 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: "map_inc": 4, 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_sessions": 12, 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_session_open": 12, 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_session_close": 0, 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_laggy": 0, 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_wr": 5612, 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_rd": 2, 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_del": 1488 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: "oft": { 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_objs": 1, 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_kv_pairs": 0, 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_updates": 0, 2024-03-18T00:07:09.343 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_removes": 0 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd-slow-ops": { 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: "slow_ops_count": 0 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: "purge_queue": { 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_ops": 0, 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_ops_high_water": 17, 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing": 0, 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_high_water": 5, 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executed_ops": 98147, 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executed": 48521, 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_item_in_journal": 0 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T00:07:09.344 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 470291, 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 112526243, 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 470291, 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 470291, 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 112526243, 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.345 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.346 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:09.346 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.346 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-objecter_bytes": { 2024-03-18T00:07:09.346 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:09.346 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2024-03-18T00:07:09.346 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:09.346 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2024-03-18T00:07:09.346 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2024-03-18T00:07:09.346 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:09.346 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:09.346 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 142184, 2024-03-18T00:07:09.346 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 666539062, 2024-03-18T00:07:09.346 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 142184, 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 666539062, 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-objecter_ops": { 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 1024, 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2024-03-18T00:07:09.347 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 142184, 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 142184, 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 142184, 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 142184, 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-write_buf_throttle": { 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:09.348 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 3758096384, 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 48521, 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 5288589, 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 48521, 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 1126, 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 5288589, 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.349 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-write_buf_throttle-0x563513a74500": { 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 3758096384, 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 265375, 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 592889412, 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 265375, 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 2282, 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 592889412, 2024-03-18T00:07:09.350 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:09.351 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:09.351 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:09.351 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:09.351 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:09.351 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:09.351 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-03-18T00:07:09.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:09 smithi012 ceph-mon[31541]: pgmap v691: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:07:10.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:09 smithi079 ceph-mon[36241]: pgmap v691: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:07:10.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:09 smithi191 ceph-mon[36708]: pgmap v691: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:07:10.162 INFO:tasks.check_counter:Daemon mds.h mds.exported=14 2024-03-18T00:07:10.163 INFO:tasks.check_counter:Daemon mds.h mds.imported=98 2024-03-18T00:07:10.163 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=4910 2024-03-18T00:07:10.163 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=15 2024-03-18T00:07:10.163 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=155 2024-03-18T00:07:10.163 DEBUG:tasks.check_counter:Getting stats from k 2024-03-18T00:07:10.163 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-18T00:07:10.420 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T00:07:11.893 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:11 smithi191 ceph-mon[36708]: pgmap v692: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T00:07:11.913 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:11 smithi079 ceph-mon[36241]: pgmap v692: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T00:07:11.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:11 smithi012 ceph-mon[31541]: pgmap v692: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T00:07:12.346 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-03-18T00:07:12.346 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T00:07:12.346 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 1099, 2024-03-18T00:07:12.346 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 1099, 2024-03-18T00:07:12.346 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 4534057, 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 348659, 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 6, 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 2, 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.081622107, 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.026088942, 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.023155782, 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.025586528, 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 1098, 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.029627090, 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000026982 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.347 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.348 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.348 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.348 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-18T00:07:12.348 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T00:07:12.348 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.348 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T00:07:12.348 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 770, 2024-03-18T00:07:12.348 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 745, 2024-03-18T00:07:12.348 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 5231414, 2024-03-18T00:07:12.348 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 297734, 2024-03-18T00:07:12.348 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 4, 2024-03-18T00:07:12.348 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 3, 2024-03-18T00:07:12.348 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.066010111, 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.018437929, 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.031714154, 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.017049283, 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 744, 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.022186187, 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000029820 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.349 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 903168, 2024-03-18T00:07:12.350 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 130080 2024-03-18T00:07:12.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.350 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T00:07:12.350 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 314, 2024-03-18T00:07:12.350 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 472, 2024-03-18T00:07:12.350 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 2453250, 2024-03-18T00:07:12.350 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 442414, 2024-03-18T00:07:12.350 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 5, 2024-03-18T00:07:12.350 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 4, 2024-03-18T00:07:12.350 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.029288179, 2024-03-18T00:07:12.350 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.010888134, 2024-03-18T00:07:12.350 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.007783310, 2024-03-18T00:07:12.350 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.007938271, 2024-03-18T00:07:12.351 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:12.351 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 470, 2024-03-18T00:07:12.351 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.021683985, 2024-03-18T00:07:12.351 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000046136 2024-03-18T00:07:12.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.351 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:12.351 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.351 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.351 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.351 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T00:07:12.351 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 342880 2024-03-18T00:07:12.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: "cct": { 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: "total_workers": 1, 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: "unhealthy_workers": 0 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: "finisher-MDSRank": { 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: "queue_len": 0, 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: "complete_latency": { 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 3379, 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.074701635, 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000022107 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: "finisher-PurgeQueue": { 2024-03-18T00:07:12.352 INFO:teuthology.orchestra.run.smithi079.stdout: "queue_len": 0, 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: "complete_latency": { 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: "mds": { 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: "request": 0, 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: "reply": 0, 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: "reply_latency": { 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.353 INFO:teuthology.orchestra.run.smithi079.stdout: "slow_reply": 0, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "forward": 0, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_fetch_complete": 0, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_fetch_keys": 0, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_commit": 0, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_split": 0, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_merge": 0, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes": 1334, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_top": 0, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_bottom": 1334, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_pin_tail": 0, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_pinned": 1334, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_expired": 0, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_with_caps": 0, 2024-03-18T00:07:12.354 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": 0, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "subtrees": 1, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse": 0, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_hit": 0, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_forward": 0, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_discover": 0, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_dir_fetch": 0, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_remote_ino": 0, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_lock": 0, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "load_cent": 0, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "q": 0, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "exported": 0, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "exported_inodes": 0, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "imported": 0, 2024-03-18T00:07:12.355 INFO:teuthology.orchestra.run.smithi079.stdout: "imported_inodes": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_dir_fetch": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_backtrace_fetch": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_peer_discover": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rfiles": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rbytes": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rsnaps": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_set_tag": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_inotable_repaired": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dir_inodes": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T00:07:12.356 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_file_inodes": 0, 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_inode_file_caps": 0, 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_grant": 0, 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_caps": 0, 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_caps_dirty": 0, 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_cap_release": 0, 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "process_request_cap_release": 0 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache": { 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_update": 0, 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_update_receipt": 0, 2024-03-18T00:07:12.357 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_try_discover": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_send_discover": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_handle_discover": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays_delayed": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays_enqueuing": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_created": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_enqueued": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_reintegrated": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_migrated": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_processing": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_enqueued": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_prioritized": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_started": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_completed": 0, 2024-03-18T00:07:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_quiesce_path": 0, 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_quiesce_inode": 0, 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_exportdir": 0, 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_flush": 0, 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_fragmentdir": 0, 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_fragstats": 0, 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_inodestats": 0 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log": { 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "evlrg": 0, 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "evadd": 0, 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "evex": 0, 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "evtrm": 0, 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "ev": 7453, 2024-03-18T00:07:12.359 INFO:teuthology.orchestra.run.smithi079.stdout: "evexg": 0, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "evexd": 0, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "segadd": 0, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "segex": 0, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "segtrm": 0, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "seg": 10, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "segmjr": 2, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "segexg": 0, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "segexd": 0, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "expos": 4194304, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "wrpos": 14956934, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rdpos": 14955796, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "jlat": { 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.360 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: "replayed": 7453 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_mem": { 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: "ino": 12, 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: "ino+": 1361, 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: "ino-": 1349, 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: "dir": 10, 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: "dir+": 76, 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: "dir-": 66, 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: "dn": 1334, 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: "dn+": 3011, 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: "dn-": 1677, 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: "cap": 0, 2024-03-18T00:07:12.361 INFO:teuthology.orchestra.run.smithi079.stdout: "cap+": 0, 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: "cap-": 0, 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: "rss": 52792, 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: "heap": 215420 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_server": { 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: "dispatch_client_request": 0, 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: "dispatch_server_request": 0, 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_request": 0, 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_session": 0, 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_peer_request": 0, 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: "req_create_latency": { 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.362 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getattr_latency": { 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getfilelock_latency": { 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: "req_link_latency": { 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookup_latency": { 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookuphash_latency": { 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupino_latency": { 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.364 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupname_latency": { 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupparent_latency": { 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupsnap_latency": { 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.366 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lssnap_latency": { 2024-03-18T00:07:12.366 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.366 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.366 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.366 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.366 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mkdir_latency": { 2024-03-18T00:07:12.366 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.366 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.366 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mknod_latency": { 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mksnap_latency": { 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: "req_open_latency": { 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.367 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: "req_readdir_latency": { 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rename_latency": { 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: "req_renamesnap_latency": { 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.368 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: "req_snapdiff_latency": { 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmdir_latency": { 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmsnap_latency": { 2024-03-18T00:07:12.369 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmxattr_latency": { 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setattr_latency": { 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setdirlayout_latency": { 2024-03-18T00:07:12.370 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setfilelock_latency": { 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setlayout_latency": { 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.371 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setxattr_latency": { 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: "req_symlink_latency": { 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: "req_unlink_latency": { 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: "cap_revoke_eviction": 0, 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: "cap_acquisition_throttle": 0, 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getvxattr_latency": { 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_sessions": { 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: "session_count": 1, 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: "session_add": 1, 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: "session_remove": 0, 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: "sessions_open": 1, 2024-03-18T00:07:12.373 INFO:teuthology.orchestra.run.smithi079.stdout: "sessions_stale": 0, 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: "total_load": 0, 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: "average_load": 0, 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: "avg_session_uptime": 2586, 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: "mdthresh_evicted": 0 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool": { 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_bytes": 0, 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_items": 0, 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_items": 0, 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_items": 0, 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T00:07:12.374 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_items": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_bytes": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_items": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_bytes": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_items": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T00:07:12.375 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_items": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_bytes": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_items": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_bytes": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_items": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_bytes": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_items": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_items": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_items": 0, 2024-03-18T00:07:12.376 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_bytes": 198287, 2024-03-18T00:07:12.377 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_items": 57, 2024-03-18T00:07:12.377 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_bytes": 0, 2024-03-18T00:07:12.377 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_items": 0, 2024-03-18T00:07:12.377 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bytes": 0, 2024-03-18T00:07:12.377 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_items": 0, 2024-03-18T00:07:12.377 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_bytes": 0, 2024-03-18T00:07:12.377 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_items": 0, 2024-03-18T00:07:12.377 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_bytes": 0, 2024-03-18T00:07:12.377 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_items": 0, 2024-03-18T00:07:12.377 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_bytes": 8792, 2024-03-18T00:07:12.377 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_items": 168, 2024-03-18T00:07:12.377 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T00:07:12.377 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_items": 0, 2024-03-18T00:07:12.378 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_bytes": 0, 2024-03-18T00:07:12.378 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_items": 0, 2024-03-18T00:07:12.378 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_bytes": 979032, 2024-03-18T00:07:12.378 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_items": 5453, 2024-03-18T00:07:12.378 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_bytes": 0, 2024-03-18T00:07:12.378 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_items": 0, 2024-03-18T00:07:12.378 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_bytes": 0, 2024-03-18T00:07:12.378 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_items": 0 2024-03-18T00:07:12.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.378 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter": { 2024-03-18T00:07:12.378 INFO:teuthology.orchestra.run.smithi079.stdout: "op_active": 0, 2024-03-18T00:07:12.378 INFO:teuthology.orchestra.run.smithi079.stdout: "op_laggy": 0, 2024-03-18T00:07:12.378 INFO:teuthology.orchestra.run.smithi079.stdout: "op_send": 1935, 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: "op_send_bytes": 21, 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: "op_resend": 3, 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: "op_reply": 1935, 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: "op_latency": { 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 1935, 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 31.157286625, 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.016101956 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: "op_inflight": 0, 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: "oplen_avg": { 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 1935, 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 1936 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.379 INFO:teuthology.orchestra.run.smithi079.stdout: "op": 1935, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "op_r": 1935, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "op_w": 0, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "op_rmw": 0, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "op_pg": 0, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_stat": 1140, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_create": 0, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_read": 793, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_write": 0, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_writefull": 0, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_writesame": 0, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_append": 0, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_zero": 0, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_truncate": 0, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_delete": 0, 2024-03-18T00:07:12.380 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_mapext": 0, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_sparse_read": 0, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_clonerange": 0, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_getxattr": 1, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_setxattr": 0, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_cmpxattr": 0, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_rmxattr": 0, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_resetxattrs": 0, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_call": 0, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_watch": 0, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_notify": 0, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_pgls": 0, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_pgls_filter": 0, 2024-03-18T00:07:12.381 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_other": 0, 2024-03-18T00:07:12.382 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_active": 0, 2024-03-18T00:07:12.382 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_send": 0, 2024-03-18T00:07:12.382 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_resend": 0, 2024-03-18T00:07:12.382 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_ping": 0, 2024-03-18T00:07:12.382 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_active": 0, 2024-03-18T00:07:12.382 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_send": 0, 2024-03-18T00:07:12.382 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_resend": 0, 2024-03-18T00:07:12.382 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_active": 0, 2024-03-18T00:07:12.382 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_send": 0, 2024-03-18T00:07:12.382 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_resend": 0, 2024-03-18T00:07:12.382 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_active": 0, 2024-03-18T00:07:12.382 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_send": 0, 2024-03-18T00:07:12.382 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_resend": 0, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "command_active": 0, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "command_send": 0, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "command_resend": 0, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "map_epoch": 94, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "map_full": 0, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "map_inc": 4, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_sessions": 6, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_session_open": 6, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_session_close": 0, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_laggy": 0, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_wr": 0, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_rd": 2, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_del": 0 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.383 INFO:teuthology.orchestra.run.smithi079.stdout: "oft": { 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_objs": 0, 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_kv_pairs": 0, 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_updates": 0, 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_removes": 0 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: "osd-slow-ops": { 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: "slow_ops_count": 0 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: "purge_queue": { 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_ops": 0, 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing": 0, 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_high_water": 0, 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executed_ops": 0, 2024-03-18T00:07:12.384 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executed": 0, 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_item_in_journal": 0 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 2179, 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 12007171, 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 2179, 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 2179, 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 12007171, 2024-03-18T00:07:12.385 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:12.386 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.386 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.386 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.386 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:12.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.386 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-objecter_bytes": { 2024-03-18T00:07:12.386 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:12.386 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2024-03-18T00:07:12.386 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:12.386 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2024-03-18T00:07:12.386 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2024-03-18T00:07:12.386 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:12.386 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 1935, 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 10763405, 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 1935, 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 10763405, 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-objecter_ops": { 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 1024, 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:12.387 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 1935, 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 1935, 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 1935, 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 1935, 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-write_buf_throttle": { 2024-03-18T00:07:12.388 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 3758096384, 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 0, 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 0, 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.389 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-write_buf_throttle-0x55638bd826e0": { 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 3758096384, 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 0, 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 0, 2024-03-18T00:07:12.390 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-03-18T00:07:12.391 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-03-18T00:07:12.391 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-03-18T00:07:12.391 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-03-18T00:07:12.391 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:12.391 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-03-18T00:07:12.391 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-03-18T00:07:13.341 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-18T00:07:13.341 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-18T00:07:13.341 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-03-18T00:07:13.341 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-03-18T00:07:13.341 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-03-18T00:07:13.342 DEBUG:tasks.check_counter:Getting stats from c 2024-03-18T00:07:13.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-18T00:07:13.597 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:07:13.914 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:13 smithi191 ceph-mon[36708]: pgmap v693: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-03-18T00:07:13.914 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:13 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:07:13.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:13 smithi012 ceph-mon[31541]: pgmap v693: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-03-18T00:07:13.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:13 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:07:14.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:13 smithi079 ceph-mon[36241]: pgmap v693: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-03-18T00:07:14.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:13 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:07:14.637 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:14 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:07:14.915 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:14 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:07:14.915 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:14 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:07:14.915 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:14 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:07:14.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:14 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:07:14.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:14 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:07:14.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:14 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:07:15.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:14 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:07:15.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:14 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' 2024-03-18T00:07:15.575 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-18T00:07:15.575 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T00:07:15.575 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 1881, 2024-03-18T00:07:15.575 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 1530, 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 694002, 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 4119830, 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 10, 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 5, 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.118994756, 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.044316521, 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.067042867, 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.015197112, 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1528, 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.070997152, 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000046464 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.576 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 2434, 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000394224, 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000161 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 20306, 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 17986, 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 8465713, 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 6302221, 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 14, 2024-03-18T00:07:15.577 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 8, 2024-03-18T00:07:15.578 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.936072053, 2024-03-18T00:07:15.578 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.362208114, 2024-03-18T00:07:15.578 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.754214005, 2024-03-18T00:07:15.578 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.008567447, 2024-03-18T00:07:15.578 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:15.578 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 17983, 2024-03-18T00:07:15.578 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.874282642, 2024-03-18T00:07:15.578 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000048617 2024-03-18T00:07:15.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.578 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:15.578 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 34023, 2024-03-18T00:07:15.578 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.004733220, 2024-03-18T00:07:15.578 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000139 2024-03-18T00:07:15.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.579 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T00:07:15.579 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 422256 2024-03-18T00:07:15.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.579 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T00:07:15.579 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 901, 2024-03-18T00:07:15.579 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 793, 2024-03-18T00:07:15.579 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 1142169, 2024-03-18T00:07:15.579 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 519177, 2024-03-18T00:07:15.579 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 7, 2024-03-18T00:07:15.579 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 6, 2024-03-18T00:07:15.580 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.074751564, 2024-03-18T00:07:15.580 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.026366212, 2024-03-18T00:07:15.580 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.040879730, 2024-03-18T00:07:15.580 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.014577505, 2024-03-18T00:07:15.580 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:15.580 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 793, 2024-03-18T00:07:15.580 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.044985604, 2024-03-18T00:07:15.580 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000056728 2024-03-18T00:07:15.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.580 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:15.580 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 377, 2024-03-18T00:07:15.581 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000059942, 2024-03-18T00:07:15.581 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000158 2024-03-18T00:07:15.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.581 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 917248, 2024-03-18T00:07:15.581 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 132944 2024-03-18T00:07:15.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.581 INFO:teuthology.orchestra.run.smithi191.stdout: "cct": { 2024-03-18T00:07:15.581 INFO:teuthology.orchestra.run.smithi191.stdout: "total_workers": 1, 2024-03-18T00:07:15.581 INFO:teuthology.orchestra.run.smithi191.stdout: "unhealthy_workers": 0 2024-03-18T00:07:15.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.581 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-MDSRank": { 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 416, 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.902634471, 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.002169794 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-PurgeQueue": { 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 874, 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.078509920, 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000089828 2024-03-18T00:07:15.582 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:15.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.583 INFO:teuthology.orchestra.run.smithi191.stdout: "mds": { 2024-03-18T00:07:15.583 INFO:teuthology.orchestra.run.smithi191.stdout: "request": 2709, 2024-03-18T00:07:15.583 INFO:teuthology.orchestra.run.smithi191.stdout: "reply": 2592, 2024-03-18T00:07:15.583 INFO:teuthology.orchestra.run.smithi191.stdout: "reply_latency": { 2024-03-18T00:07:15.583 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 2592, 2024-03-18T00:07:15.583 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 74.587992457, 2024-03-18T00:07:15.583 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.028776231 2024-03-18T00:07:15.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.583 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_reply": 0, 2024-03-18T00:07:15.583 INFO:teuthology.orchestra.run.smithi191.stdout: "forward": 117, 2024-03-18T00:07:15.583 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_complete": 0, 2024-03-18T00:07:15.583 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_keys": 0, 2024-03-18T00:07:15.584 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_commit": 29, 2024-03-18T00:07:15.584 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_split": 2, 2024-03-18T00:07:15.584 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_merge": 0, 2024-03-18T00:07:15.584 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes": 27, 2024-03-18T00:07:15.584 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_top": 8, 2024-03-18T00:07:15.584 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_bottom": 9, 2024-03-18T00:07:15.584 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pin_tail": 10, 2024-03-18T00:07:15.584 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pinned": 13, 2024-03-18T00:07:15.584 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_expired": 402, 2024-03-18T00:07:15.584 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_with_caps": 0, 2024-03-18T00:07:15.584 INFO:teuthology.orchestra.run.smithi191.stdout: "caps": 0, 2024-03-18T00:07:15.584 INFO:teuthology.orchestra.run.smithi191.stdout: "subtrees": 4, 2024-03-18T00:07:15.585 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse": 8864, 2024-03-18T00:07:15.585 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_hit": 7162, 2024-03-18T00:07:15.585 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_forward": 117, 2024-03-18T00:07:15.585 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_discover": 200, 2024-03-18T00:07:15.585 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_dir_fetch": 0, 2024-03-18T00:07:15.585 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_remote_ino": 0, 2024-03-18T00:07:15.585 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_lock": 54, 2024-03-18T00:07:15.585 INFO:teuthology.orchestra.run.smithi191.stdout: "load_cent": 0, 2024-03-18T00:07:15.585 INFO:teuthology.orchestra.run.smithi191.stdout: "q": 0, 2024-03-18T00:07:15.585 INFO:teuthology.orchestra.run.smithi191.stdout: "exported": 16, 2024-03-18T00:07:15.585 INFO:teuthology.orchestra.run.smithi191.stdout: "exported_inodes": 3, 2024-03-18T00:07:15.585 INFO:teuthology.orchestra.run.smithi191.stdout: "imported": 2, 2024-03-18T00:07:15.585 INFO:teuthology.orchestra.run.smithi191.stdout: "imported_inodes": 318, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_dir_fetch": 0, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_backtrace_fetch": 0, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_peer_discover": 0, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rfiles": 2278, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rbytes": 28871204, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rsnaps": 0, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_set_tag": 0, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_inotable_repaired": 0, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_inodes": 0, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T00:07:15.586 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_file_inodes": 0, 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_inode_file_caps": 33, 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_revoke": 513, 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_grant": 837, 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps": 931, 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps_dirty": 412, 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_cap_release": 27, 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "process_request_cap_release": 1591 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_cache": { 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update": 166, 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update_receipt": 4759, 2024-03-18T00:07:15.587 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_try_discover": 3156, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_send_discover": 3284, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_handle_discover": 73, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays": 0, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_delayed": 0, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_enqueuing": 0, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_created": 412, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_enqueued": 412, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_reintegrated": 0, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_migrated": 0, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_processing": 0, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_enqueued": 0, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_prioritized": 0, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_started": 0, 2024-03-18T00:07:15.588 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_completed": 0, 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_path": 0, 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_inode": 0, 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_exportdir": 16, 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_flush": 0, 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragmentdir": 2, 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragstats": 0, 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_inodestats": 0 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_log": { 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: "evlrg": 0, 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: "evadd": 2385, 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: "evex": 0, 2024-03-18T00:07:15.589 INFO:teuthology.orchestra.run.smithi191.stdout: "evtrm": 0, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "ev": 2385, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "evexg": 0, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "evexd": 0, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "segadd": 4, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "segex": 0, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "segtrm": 0, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "seg": 4, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "segmjr": 2, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "segexg": 0, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "segexd": 0, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "expos": 4194304, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "wrpos": 7597654, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "rdpos": 0, 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "jlat": { 2024-03-18T00:07:15.590 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 125, 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 3.845294352, 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.030762354 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: "replayed": 0 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_mem": { 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: "ino": 32, 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: "ino+": 645, 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: "ino-": 613, 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: "dir": 31, 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: "dir+": 1453, 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: "dir-": 1422, 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: "dn": 27, 2024-03-18T00:07:15.591 INFO:teuthology.orchestra.run.smithi191.stdout: "dn+": 1585, 2024-03-18T00:07:15.592 INFO:teuthology.orchestra.run.smithi191.stdout: "dn-": 1558, 2024-03-18T00:07:15.592 INFO:teuthology.orchestra.run.smithi191.stdout: "cap": 0, 2024-03-18T00:07:15.592 INFO:teuthology.orchestra.run.smithi191.stdout: "cap+": 774, 2024-03-18T00:07:15.592 INFO:teuthology.orchestra.run.smithi191.stdout: "cap-": 774, 2024-03-18T00:07:15.592 INFO:teuthology.orchestra.run.smithi191.stdout: "rss": 61748, 2024-03-18T00:07:15.592 INFO:teuthology.orchestra.run.smithi191.stdout: "heap": 223612 2024-03-18T00:07:15.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.592 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_server": { 2024-03-18T00:07:15.592 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_client_request": 3759, 2024-03-18T00:07:15.592 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_server_request": 170, 2024-03-18T00:07:15.592 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_request": 2709, 2024-03-18T00:07:15.592 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_session": 7, 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_peer_request": 1259, 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: "req_create_latency": { 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 268, 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 2.416259265, 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.009015892 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getattr_latency": { 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getfilelock_latency": { 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: "req_link_latency": { 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookup_latency": { 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1432, 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 65.421046009, 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.045685087 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookuphash_latency": { 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.594 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupino_latency": { 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupname_latency": { 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupparent_latency": { 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.596 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupsnap_latency": { 2024-03-18T00:07:15.596 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.596 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.596 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.596 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lssnap_latency": { 2024-03-18T00:07:15.596 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.597 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.597 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.597 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mkdir_latency": { 2024-03-18T00:07:15.597 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.597 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.597 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.597 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mknod_latency": { 2024-03-18T00:07:15.597 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.597 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.597 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.598 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mksnap_latency": { 2024-03-18T00:07:15.598 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.598 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.598 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.598 INFO:teuthology.orchestra.run.smithi191.stdout: "req_open_latency": { 2024-03-18T00:07:15.598 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 250, 2024-03-18T00:07:15.598 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.201971261, 2024-03-18T00:07:15.598 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000807885 2024-03-18T00:07:15.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.598 INFO:teuthology.orchestra.run.smithi191.stdout: "req_readdir_latency": { 2024-03-18T00:07:15.598 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 16, 2024-03-18T00:07:15.599 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.019805578, 2024-03-18T00:07:15.599 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001237848 2024-03-18T00:07:15.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.599 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rename_latency": { 2024-03-18T00:07:15.599 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 270, 2024-03-18T00:07:15.599 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 5.485957218, 2024-03-18T00:07:15.599 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.020318360 2024-03-18T00:07:15.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.599 INFO:teuthology.orchestra.run.smithi191.stdout: "req_renamesnap_latency": { 2024-03-18T00:07:15.599 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.599 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.599 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: "req_snapdiff_latency": { 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmdir_latency": { 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmsnap_latency": { 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.600 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmxattr_latency": { 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setattr_latency": { 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setdirlayout_latency": { 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.601 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.602 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.602 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setfilelock_latency": { 2024-03-18T00:07:15.602 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.602 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.602 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.602 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setlayout_latency": { 2024-03-18T00:07:15.602 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.602 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.602 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.602 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setxattr_latency": { 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: "req_symlink_latency": { 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: "req_unlink_latency": { 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 356, 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 1.042953126, 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.002929643 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_revoke_eviction": 0, 2024-03-18T00:07:15.603 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_acquisition_throttle": 0, 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getvxattr_latency": { 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_sessions": { 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: "session_count": 1, 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: "session_add": 1, 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: "session_remove": 0, 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_open": 1, 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_stale": 0, 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: "total_load": 690, 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: "average_load": 690, 2024-03-18T00:07:15.604 INFO:teuthology.orchestra.run.smithi191.stdout: "avg_session_uptime": 409, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "mdthresh_evicted": 0 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "mempool": { 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_bytes": 0, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_items": 0, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_items": 0, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_items": 0, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_items": 0, 2024-03-18T00:07:15.605 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_bytes": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_items": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_bytes": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_items": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_items": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_bytes": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_items": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T00:07:15.606 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_bytes": 0, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_items": 0, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_bytes": 0, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_items": 0, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_items": 0, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_items": 0, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_bytes": 310721, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_items": 137, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_bytes": 704, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_items": 8, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_bytes": 0, 2024-03-18T00:07:15.607 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_items": 0, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_bytes": 0, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_items": 0, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_bytes": 0, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_items": 0, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_bytes": 8792, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_items": 168, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_items": 0, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_bytes": 0, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_items": 0, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_bytes": 130357, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_items": 485, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_bytes": 0, 2024-03-18T00:07:15.608 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_items": 0, 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_bytes": 0, 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_items": 0 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: "objecter": { 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: "op_active": 0, 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: "op_laggy": 0, 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send": 1182, 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send_bytes": 3823535, 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: "op_resend": 29, 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: "op_reply": 1182, 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: "op_latency": { 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1182, 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 24.275579612, 2024-03-18T00:07:15.609 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.020537715 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: "op_inflight": 0, 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: "oplen_avg": { 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1182, 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 1321 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: "op": 1182, 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: "op_r": 24, 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: "op_w": 1158, 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: "op_rmw": 0, 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: "op_pg": 0, 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_stat": 0, 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_create": 10, 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_read": 24, 2024-03-18T00:07:15.610 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_write": 149, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writefull": 43, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writesame": 0, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_append": 0, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_zero": 0, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_truncate": 2, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_delete": 836, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_mapext": 0, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_sparse_read": 0, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_clonerange": 0, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_getxattr": 0, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_setxattr": 20, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_cmpxattr": 0, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_rmxattr": 0, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_resetxattrs": 0, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_call": 0, 2024-03-18T00:07:15.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_watch": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_notify": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls_filter": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_other": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_active": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_send": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_resend": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_ping": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_active": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_send": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_resend": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_active": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_send": 0, 2024-03-18T00:07:15.612 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_resend": 0, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_active": 0, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_send": 0, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_resend": 0, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "command_active": 0, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "command_send": 0, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "command_resend": 0, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "map_epoch": 94, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "map_full": 0, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "map_inc": 4, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_sessions": 12, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_open": 12, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_close": 0, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_laggy": 0, 2024-03-18T00:07:15.613 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_wr": 195, 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_rd": 0, 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_del": 42 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: "oft": { 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_objs": 1, 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_kv_pairs": 0, 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_updates": 0, 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_removes": 0 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: "osd-slow-ops": { 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_ops_count": 0 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: "purge_queue": { 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops": 0, 2024-03-18T00:07:15.614 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops_high_water": 10, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing": 0, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_high_water": 5, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed_ops": 824, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed": 412, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_item_in_journal": 0 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 23084, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 8207709, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 23084, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T00:07:15.615 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 23084, 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 8207709, 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_bytes": { 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T00:07:15.616 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 1182, 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 3868443, 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 1182, 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 3868443, 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_ops": { 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 1024, 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:15.617 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 1182, 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 1182, 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 1182, 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 1182, 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle": { 2024-03-18T00:07:15.618 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 412, 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 44908, 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 412, 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 24, 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 44908, 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:15.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle-0x55e1dfefa500": { 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 2385, 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 3403350, 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 2385, 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 125, 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 3403350, 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:15.620 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:15.621 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:15.621 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:15.621 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:15.621 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-18T00:07:15.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:15 smithi079 ceph-mon[36241]: pgmap v694: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T00:07:15.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:15 smithi191 ceph-mon[36708]: pgmap v694: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T00:07:15.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:15 smithi012 ceph-mon[31541]: pgmap v694: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T00:07:16.474 INFO:tasks.check_counter:Daemon mds.c mds.exported=16 2024-03-18T00:07:16.475 INFO:tasks.check_counter:Daemon mds.c mds.imported=2 2024-03-18T00:07:16.475 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=166 2024-03-18T00:07:16.475 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=4759 2024-03-18T00:07:16.475 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=2 2024-03-18T00:07:16.475 DEBUG:tasks.check_counter:Getting stats from f 2024-03-18T00:07:16.475 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-18T00:07:16.761 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:07:17.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:17 smithi191 ceph-mon[36708]: pgmap v695: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-18T00:07:17.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:17 smithi012 ceph-mon[31541]: pgmap v695: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-18T00:07:18.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:17 smithi079 ceph-mon[36241]: pgmap v695: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-18T00:07:18.558 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-18T00:07:18.559 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T00:07:18.559 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 2109, 2024-03-18T00:07:18.559 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 2111, 2024-03-18T00:07:18.559 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 9626288, 2024-03-18T00:07:18.559 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 670669, 2024-03-18T00:07:18.559 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 6, 2024-03-18T00:07:18.559 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 2, 2024-03-18T00:07:18.559 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.137494946, 2024-03-18T00:07:18.559 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.039200625, 2024-03-18T00:07:18.559 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.039501209, 2024-03-18T00:07:18.559 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.049311581, 2024-03-18T00:07:18.559 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:18.560 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 2109, 2024-03-18T00:07:18.560 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.051657144, 2024-03-18T00:07:18.560 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000024493 2024-03-18T00:07:18.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.560 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:18.560 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.560 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.560 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.560 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T00:07:18.560 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T00:07:18.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.560 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 72, 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 228, 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 17690, 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 366784, 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 5, 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 4, 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.012028971, 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.006326583, 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.002129166, 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.002092803, 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 226, 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.011146647, 2024-03-18T00:07:18.561 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000049321 2024-03-18T00:07:18.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.562 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:18.562 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.562 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.562 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.562 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T00:07:18.562 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 344144 2024-03-18T00:07:18.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.562 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T00:07:18.562 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 246, 2024-03-18T00:07:18.562 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 219, 2024-03-18T00:07:18.562 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 905964, 2024-03-18T00:07:18.563 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 129566, 2024-03-18T00:07:18.563 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 3, 2024-03-18T00:07:18.563 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 2, 2024-03-18T00:07:18.563 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.028891609, 2024-03-18T00:07:18.563 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.008673022, 2024-03-18T00:07:18.563 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.020200600, 2024-03-18T00:07:18.563 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.004144302, 2024-03-18T00:07:18.563 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:18.563 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 219, 2024-03-18T00:07:18.563 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.007563336, 2024-03-18T00:07:18.563 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000034535 2024-03-18T00:07:18.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.563 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 905696, 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 129248 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: "cct": { 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: "total_workers": 1, 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: "unhealthy_workers": 0 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-MDSRank": { 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T00:07:18.564 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 3719, 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.075989012, 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000020432 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-PurgeQueue": { 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.565 INFO:teuthology.orchestra.run.smithi191.stdout: "mds": { 2024-03-18T00:07:18.566 INFO:teuthology.orchestra.run.smithi191.stdout: "request": 0, 2024-03-18T00:07:18.566 INFO:teuthology.orchestra.run.smithi191.stdout: "reply": 0, 2024-03-18T00:07:18.566 INFO:teuthology.orchestra.run.smithi191.stdout: "reply_latency": { 2024-03-18T00:07:18.566 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.566 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.566 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.566 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_reply": 0, 2024-03-18T00:07:18.566 INFO:teuthology.orchestra.run.smithi191.stdout: "forward": 0, 2024-03-18T00:07:18.566 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_complete": 0, 2024-03-18T00:07:18.566 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_keys": 0, 2024-03-18T00:07:18.566 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_commit": 0, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_split": 0, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_merge": 0, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes": 1184, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_top": 0, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_bottom": 1184, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pin_tail": 0, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pinned": 1184, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_expired": 0, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_with_caps": 0, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "caps": 0, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "subtrees": 1, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse": 0, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_hit": 0, 2024-03-18T00:07:18.567 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_forward": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_discover": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_dir_fetch": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_remote_ino": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_lock": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "load_cent": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "q": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "exported": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "exported_inodes": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "imported": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "imported_inodes": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_dir_fetch": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_backtrace_fetch": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_peer_discover": 0, 2024-03-18T00:07:18.568 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rfiles": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rbytes": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rsnaps": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_set_tag": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_inotable_repaired": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_inodes": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_file_inodes": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_inode_file_caps": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_grant": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T00:07:18.569 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T00:07:18.570 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T00:07:18.570 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps": 0, 2024-03-18T00:07:18.570 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps_dirty": 0, 2024-03-18T00:07:18.570 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_cap_release": 0, 2024-03-18T00:07:18.570 INFO:teuthology.orchestra.run.smithi191.stdout: "process_request_cap_release": 0 2024-03-18T00:07:18.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.570 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_cache": { 2024-03-18T00:07:18.570 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update": 0, 2024-03-18T00:07:18.571 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update_receipt": 0, 2024-03-18T00:07:18.571 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_try_discover": 0, 2024-03-18T00:07:18.571 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_send_discover": 0, 2024-03-18T00:07:18.571 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_handle_discover": 0, 2024-03-18T00:07:18.571 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays": 0, 2024-03-18T00:07:18.571 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_delayed": 0, 2024-03-18T00:07:18.571 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_enqueuing": 0, 2024-03-18T00:07:18.572 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_created": 0, 2024-03-18T00:07:18.572 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_enqueued": 0, 2024-03-18T00:07:18.572 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_reintegrated": 0, 2024-03-18T00:07:18.572 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_migrated": 0, 2024-03-18T00:07:18.572 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_processing": 0, 2024-03-18T00:07:18.572 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_enqueued": 0, 2024-03-18T00:07:18.572 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_prioritized": 0, 2024-03-18T00:07:18.572 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_started": 0, 2024-03-18T00:07:18.572 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_completed": 0, 2024-03-18T00:07:18.572 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_path": 0, 2024-03-18T00:07:18.572 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_inode": 0, 2024-03-18T00:07:18.572 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T00:07:18.572 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_exportdir": 0, 2024-03-18T00:07:18.573 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_flush": 0, 2024-03-18T00:07:18.573 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragmentdir": 0, 2024-03-18T00:07:18.573 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragstats": 0, 2024-03-18T00:07:18.573 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_inodestats": 0 2024-03-18T00:07:18.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.573 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_log": { 2024-03-18T00:07:18.573 INFO:teuthology.orchestra.run.smithi191.stdout: "evlrg": 0, 2024-03-18T00:07:18.573 INFO:teuthology.orchestra.run.smithi191.stdout: "evadd": 0, 2024-03-18T00:07:18.574 INFO:teuthology.orchestra.run.smithi191.stdout: "evex": 0, 2024-03-18T00:07:18.574 INFO:teuthology.orchestra.run.smithi191.stdout: "evtrm": 0, 2024-03-18T00:07:18.574 INFO:teuthology.orchestra.run.smithi191.stdout: "ev": 6234, 2024-03-18T00:07:18.574 INFO:teuthology.orchestra.run.smithi191.stdout: "evexg": 0, 2024-03-18T00:07:18.574 INFO:teuthology.orchestra.run.smithi191.stdout: "evexd": 0, 2024-03-18T00:07:18.574 INFO:teuthology.orchestra.run.smithi191.stdout: "segadd": 0, 2024-03-18T00:07:18.574 INFO:teuthology.orchestra.run.smithi191.stdout: "segex": 0, 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: "segtrm": 0, 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: "seg": 8, 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: "segmjr": 2, 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: "segexg": 0, 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: "segexd": 0, 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: "expos": 4194304, 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: "wrpos": 13225601, 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: "rdpos": 13224463, 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: "jlat": { 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.575 INFO:teuthology.orchestra.run.smithi191.stdout: "replayed": 6234 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_mem": { 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: "ino": 10, 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: "ino+": 1208, 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: "ino-": 1198, 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: "dir": 8, 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: "dir+": 71, 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: "dir-": 63, 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: "dn": 1184, 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: "dn+": 2536, 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: "dn-": 1352, 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: "cap": 0, 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: "cap+": 0, 2024-03-18T00:07:18.576 INFO:teuthology.orchestra.run.smithi191.stdout: "cap-": 0, 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: "rss": 54264, 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: "heap": 215420 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_server": { 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_client_request": 0, 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_server_request": 0, 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_request": 0, 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_session": 0, 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_peer_request": 0, 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: "req_create_latency": { 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.578 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getattr_latency": { 2024-03-18T00:07:18.578 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.578 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.578 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.578 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getfilelock_latency": { 2024-03-18T00:07:18.578 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.578 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.578 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.578 INFO:teuthology.orchestra.run.smithi191.stdout: "req_link_latency": { 2024-03-18T00:07:18.578 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.579 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.579 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.579 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookup_latency": { 2024-03-18T00:07:18.579 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.579 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.579 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.579 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookuphash_latency": { 2024-03-18T00:07:18.579 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.579 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.579 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupino_latency": { 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupname_latency": { 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupparent_latency": { 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.581 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupsnap_latency": { 2024-03-18T00:07:18.581 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.581 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.581 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.581 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lssnap_latency": { 2024-03-18T00:07:18.581 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.581 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.581 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.581 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mkdir_latency": { 2024-03-18T00:07:18.582 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.582 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.582 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.582 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mknod_latency": { 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mksnap_latency": { 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: "req_open_latency": { 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: "req_readdir_latency": { 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rename_latency": { 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: "req_renamesnap_latency": { 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.584 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: "req_snapdiff_latency": { 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmdir_latency": { 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmsnap_latency": { 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.585 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmxattr_latency": { 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setattr_latency": { 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setdirlayout_latency": { 2024-03-18T00:07:18.586 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.587 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.587 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.587 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setfilelock_latency": { 2024-03-18T00:07:18.587 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.587 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.587 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.587 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setlayout_latency": { 2024-03-18T00:07:18.587 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.587 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.588 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.588 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setxattr_latency": { 2024-03-18T00:07:18.588 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.588 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.588 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.588 INFO:teuthology.orchestra.run.smithi191.stdout: "req_symlink_latency": { 2024-03-18T00:07:18.588 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.588 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.588 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.589 INFO:teuthology.orchestra.run.smithi191.stdout: "req_unlink_latency": { 2024-03-18T00:07:18.589 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.589 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.589 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.589 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_revoke_eviction": 0, 2024-03-18T00:07:18.589 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_acquisition_throttle": 0, 2024-03-18T00:07:18.589 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getvxattr_latency": { 2024-03-18T00:07:18.589 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.589 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.589 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.589 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:18.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.590 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_sessions": { 2024-03-18T00:07:18.590 INFO:teuthology.orchestra.run.smithi191.stdout: "session_count": 1, 2024-03-18T00:07:18.590 INFO:teuthology.orchestra.run.smithi191.stdout: "session_add": 2, 2024-03-18T00:07:18.590 INFO:teuthology.orchestra.run.smithi191.stdout: "session_remove": 1, 2024-03-18T00:07:18.590 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_open": 1, 2024-03-18T00:07:18.590 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_stale": 0, 2024-03-18T00:07:18.590 INFO:teuthology.orchestra.run.smithi191.stdout: "total_load": 0, 2024-03-18T00:07:18.590 INFO:teuthology.orchestra.run.smithi191.stdout: "average_load": 0, 2024-03-18T00:07:18.590 INFO:teuthology.orchestra.run.smithi191.stdout: "avg_session_uptime": 2579, 2024-03-18T00:07:18.590 INFO:teuthology.orchestra.run.smithi191.stdout: "mdthresh_evicted": 0 2024-03-18T00:07:18.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.590 INFO:teuthology.orchestra.run.smithi191.stdout: "mempool": { 2024-03-18T00:07:18.590 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_bytes": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_items": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_items": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_items": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_items": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T00:07:18.591 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_bytes": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_items": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_bytes": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_items": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_items": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_bytes": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_items": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T00:07:18.592 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_bytes": 0, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_items": 0, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_bytes": 0, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_items": 0, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_items": 0, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_items": 0, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_bytes": 190185, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_items": 53, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_bytes": 0, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_items": 0, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_bytes": 0, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_items": 0, 2024-03-18T00:07:18.593 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_bytes": 0, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_items": 0, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_bytes": 0, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_items": 0, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_bytes": 8792, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_items": 168, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_items": 0, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_bytes": 0, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_items": 0, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_bytes": 866456, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_items": 4831, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_bytes": 0, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_items": 0, 2024-03-18T00:07:18.594 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_bytes": 0, 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_items": 0 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: "objecter": { 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: "op_active": 0, 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: "op_laggy": 0, 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send": 2181, 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send_bytes": 21, 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: "op_resend": 3, 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: "op_reply": 2181, 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: "op_latency": { 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 2181, 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 2.271960280, 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001041705 2024-03-18T00:07:18.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.596 INFO:teuthology.orchestra.run.smithi191.stdout: "op_inflight": 0, 2024-03-18T00:07:18.596 INFO:teuthology.orchestra.run.smithi191.stdout: "oplen_avg": { 2024-03-18T00:07:18.596 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 2181, 2024-03-18T00:07:18.596 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 2182 2024-03-18T00:07:18.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.596 INFO:teuthology.orchestra.run.smithi191.stdout: "op": 2181, 2024-03-18T00:07:18.596 INFO:teuthology.orchestra.run.smithi191.stdout: "op_r": 2181, 2024-03-18T00:07:18.596 INFO:teuthology.orchestra.run.smithi191.stdout: "op_w": 0, 2024-03-18T00:07:18.596 INFO:teuthology.orchestra.run.smithi191.stdout: "op_rmw": 0, 2024-03-18T00:07:18.596 INFO:teuthology.orchestra.run.smithi191.stdout: "op_pg": 0, 2024-03-18T00:07:18.596 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_stat": 1332, 2024-03-18T00:07:18.596 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_create": 0, 2024-03-18T00:07:18.596 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_read": 847, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_write": 0, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writefull": 0, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writesame": 0, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_append": 0, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_zero": 0, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_truncate": 0, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_delete": 0, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_mapext": 0, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_sparse_read": 0, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_clonerange": 0, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_getxattr": 1, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_setxattr": 0, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_cmpxattr": 0, 2024-03-18T00:07:18.597 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_rmxattr": 0, 2024-03-18T00:07:18.598 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_resetxattrs": 0, 2024-03-18T00:07:18.598 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_call": 0, 2024-03-18T00:07:18.598 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_watch": 0, 2024-03-18T00:07:18.598 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_notify": 0, 2024-03-18T00:07:18.598 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T00:07:18.598 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls": 0, 2024-03-18T00:07:18.598 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls_filter": 0, 2024-03-18T00:07:18.598 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_other": 0, 2024-03-18T00:07:18.598 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_active": 0, 2024-03-18T00:07:18.598 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_send": 0, 2024-03-18T00:07:18.598 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_resend": 0, 2024-03-18T00:07:18.598 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_ping": 0, 2024-03-18T00:07:18.598 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_active": 0, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_send": 0, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_resend": 0, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_active": 0, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_send": 0, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_resend": 0, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_active": 0, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_send": 0, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_resend": 0, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "command_active": 0, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "command_send": 0, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "command_resend": 0, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "map_epoch": 94, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "map_full": 0, 2024-03-18T00:07:18.599 INFO:teuthology.orchestra.run.smithi191.stdout: "map_inc": 4, 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_sessions": 5, 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_open": 5, 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_close": 0, 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_laggy": 0, 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_wr": 0, 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_rd": 2, 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_del": 0 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: "oft": { 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_objs": 0, 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_kv_pairs": 0, 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_updates": 0, 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_removes": 0 2024-03-18T00:07:18.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.601 INFO:teuthology.orchestra.run.smithi191.stdout: "osd-slow-ops": { 2024-03-18T00:07:18.601 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_ops_count": 0 2024-03-18T00:07:18.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.601 INFO:teuthology.orchestra.run.smithi191.stdout: "purge_queue": { 2024-03-18T00:07:18.601 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops": 0, 2024-03-18T00:07:18.601 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T00:07:18.601 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing": 0, 2024-03-18T00:07:18.601 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_high_water": 0, 2024-03-18T00:07:18.601 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed_ops": 0, 2024-03-18T00:07:18.601 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed": 0, 2024-03-18T00:07:18.601 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_item_in_journal": 0 2024-03-18T00:07:18.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.602 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T00:07:18.602 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:18.602 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T00:07:18.602 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:18.602 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 2423, 2024-03-18T00:07:18.602 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 10316536, 2024-03-18T00:07:18.602 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:18.602 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 2423, 2024-03-18T00:07:18.602 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T00:07:18.603 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T00:07:18.603 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 2423, 2024-03-18T00:07:18.603 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 10316536, 2024-03-18T00:07:18.603 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:18.603 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.604 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.604 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.604 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:18.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.604 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_bytes": { 2024-03-18T00:07:18.604 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:18.604 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T00:07:18.604 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T00:07:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T00:07:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 2181, 2024-03-18T00:07:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 9031302, 2024-03-18T00:07:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 2181, 2024-03-18T00:07:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 9031302, 2024-03-18T00:07:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_ops": { 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 1024, 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 2181, 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 2181, 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 2181, 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 2181, 2024-03-18T00:07:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle": { 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:18.607 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 0, 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 0, 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle-0x5566c3cf26e0": { 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T00:07:18.608 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T00:07:18.609 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:18.609 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:18.609 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T00:07:18.609 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T00:07:18.609 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 0, 2024-03-18T00:07:18.609 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 0, 2024-03-18T00:07:18.609 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:18.609 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:18.609 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:18.609 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:18.609 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:18.609 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:18.609 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-18T00:07:19.471 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-18T00:07:19.471 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-18T00:07:19.471 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-18T00:07:19.471 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-18T00:07:19.471 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-18T00:07:19.471 DEBUG:tasks.check_counter:Getting stats from i 2024-03-18T00:07:19.472 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-18T00:07:19.724 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:07:19.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:19 smithi191 ceph-mon[36708]: pgmap v696: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:07:19.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:19 smithi012 ceph-mon[31541]: pgmap v696: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:07:20.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:19 smithi079 ceph-mon[36241]: pgmap v696: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:07:21.385 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-18T00:07:21.385 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T00:07:21.385 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 2655, 2024-03-18T00:07:21.385 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 2466, 2024-03-18T00:07:21.385 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 871053, 2024-03-18T00:07:21.385 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 7617357, 2024-03-18T00:07:21.386 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 10, 2024-03-18T00:07:21.386 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 5, 2024-03-18T00:07:21.386 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.203980639, 2024-03-18T00:07:21.386 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.076145014, 2024-03-18T00:07:21.386 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.086681376, 2024-03-18T00:07:21.386 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.045423292, 2024-03-18T00:07:21.386 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:21.386 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 2464, 2024-03-18T00:07:21.386 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.172064817, 2024-03-18T00:07:21.386 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000069831 2024-03-18T00:07:21.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.386 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:21.386 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 2763, 2024-03-18T00:07:21.387 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000495053, 2024-03-18T00:07:21.387 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000179 2024-03-18T00:07:21.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.387 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T00:07:21.387 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T00:07:21.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.387 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T00:07:21.387 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 18077, 2024-03-18T00:07:21.387 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 16518, 2024-03-18T00:07:21.387 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 7431705, 2024-03-18T00:07:21.387 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 8739121, 2024-03-18T00:07:21.387 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 9, 2024-03-18T00:07:21.387 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 7, 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.957069191, 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.369601881, 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.694326975, 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.035437429, 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 16517, 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.531716397, 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000032192 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 31403, 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.004346516, 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000138 2024-03-18T00:07:21.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 346032 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 19488, 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 25182, 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 7968688, 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 11504883, 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 12, 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 7, 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.978860920, 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.411966451, 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.738990810, 2024-03-18T00:07:21.389 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.031508305, 2024-03-18T00:07:21.390 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:21.390 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 25181, 2024-03-18T00:07:21.390 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 1.216607777, 2024-03-18T00:07:21.390 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000048314 2024-03-18T00:07:21.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.390 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:21.390 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 33582, 2024-03-18T00:07:21.390 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.005377216, 2024-03-18T00:07:21.390 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000160 2024-03-18T00:07:21.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.390 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 920976, 2024-03-18T00:07:21.390 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 132848 2024-03-18T00:07:21.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.391 INFO:teuthology.orchestra.run.smithi191.stdout: "cct": { 2024-03-18T00:07:21.391 INFO:teuthology.orchestra.run.smithi191.stdout: "total_workers": 1, 2024-03-18T00:07:21.391 INFO:teuthology.orchestra.run.smithi191.stdout: "unhealthy_workers": 0 2024-03-18T00:07:21.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.391 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-MDSRank": { 2024-03-18T00:07:21.391 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T00:07:21.391 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T00:07:21.391 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 963, 2024-03-18T00:07:21.391 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 3.350881327, 2024-03-18T00:07:21.391 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.003479627 2024-03-18T00:07:21.391 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:21.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.391 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-PurgeQueue": { 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 2749, 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.224726309, 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000081748 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: "mds": { 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: "request": 7782, 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: "reply": 7364, 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: "reply_latency": { 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 7364, 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 59.385134096, 2024-03-18T00:07:21.392 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.008064249 2024-03-18T00:07:21.393 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.393 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_reply": 0, 2024-03-18T00:07:21.393 INFO:teuthology.orchestra.run.smithi191.stdout: "forward": 418, 2024-03-18T00:07:21.393 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_complete": 0, 2024-03-18T00:07:21.393 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_keys": 0, 2024-03-18T00:07:21.393 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_commit": 47, 2024-03-18T00:07:21.393 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_split": 4, 2024-03-18T00:07:21.393 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_merge": 0, 2024-03-18T00:07:21.393 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes": 27, 2024-03-18T00:07:21.393 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_top": 8, 2024-03-18T00:07:21.394 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_bottom": 9, 2024-03-18T00:07:21.394 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pin_tail": 10, 2024-03-18T00:07:21.394 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pinned": 13, 2024-03-18T00:07:21.394 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_expired": 434, 2024-03-18T00:07:21.394 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_with_caps": 0, 2024-03-18T00:07:21.394 INFO:teuthology.orchestra.run.smithi191.stdout: "caps": 0, 2024-03-18T00:07:21.394 INFO:teuthology.orchestra.run.smithi191.stdout: "subtrees": 4, 2024-03-18T00:07:21.394 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse": 19669, 2024-03-18T00:07:21.394 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_hit": 14181, 2024-03-18T00:07:21.394 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_forward": 418, 2024-03-18T00:07:21.394 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_discover": 207, 2024-03-18T00:07:21.394 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_dir_fetch": 0, 2024-03-18T00:07:21.394 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_remote_ino": 0, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_lock": 201, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "load_cent": 0, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "q": 0, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "exported": 33, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "exported_inodes": 19, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "imported": 5, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "imported_inodes": 633, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_dir_fetch": 0, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_backtrace_fetch": 0, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_peer_discover": 0, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rfiles": 2278, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rbytes": 28871204, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rsnaps": 0, 2024-03-18T00:07:21.395 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_set_tag": 0, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_inotable_repaired": 0, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_inodes": 0, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_file_inodes": 0, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_inode_file_caps": 51, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_revoke": 1628, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_grant": 2822, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps": 3184, 2024-03-18T00:07:21.396 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps_dirty": 1326, 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_cap_release": 151, 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: "process_request_cap_release": 5311 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_cache": { 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update": 436, 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update_receipt": 4393, 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_try_discover": 2844, 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_send_discover": 2945, 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_handle_discover": 169, 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays": 0, 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_delayed": 0, 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_enqueuing": 0, 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_created": 1325, 2024-03-18T00:07:21.397 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_enqueued": 1325, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_reintegrated": 0, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_migrated": 0, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_processing": 0, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_enqueued": 0, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_prioritized": 0, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_started": 0, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_completed": 0, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_path": 0, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_inode": 0, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_exportdir": 33, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_flush": 0, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragmentdir": 4, 2024-03-18T00:07:21.398 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragstats": 0, 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_inodestats": 0 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_log": { 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "evlrg": 0, 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "evadd": 7453, 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "evex": 0, 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "evtrm": 0, 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "ev": 7453, 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "evexg": 0, 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "evexd": 0, 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "segadd": 10, 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "segex": 0, 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "segtrm": 0, 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "seg": 10, 2024-03-18T00:07:21.399 INFO:teuthology.orchestra.run.smithi191.stdout: "segmjr": 2, 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: "segexg": 0, 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: "segexd": 0, 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: "expos": 4194304, 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: "wrpos": 14956934, 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: "rdpos": 0, 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: "jlat": { 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 333, 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 4.214456012, 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.012656024 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: "replayed": 0 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_mem": { 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: "ino": 32, 2024-03-18T00:07:21.400 INFO:teuthology.orchestra.run.smithi191.stdout: "ino+": 1565, 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: "ino-": 1533, 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: "dir": 31, 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: "dir+": 1481, 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: "dir-": 1450, 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: "dn": 27, 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: "dn+": 3417, 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: "dn-": 3390, 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: "cap": 0, 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: "cap+": 2073, 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: "cap-": 2073, 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: "rss": 65792, 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: "heap": 223612 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.401 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_server": { 2024-03-18T00:07:21.402 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_client_request": 11504, 2024-03-18T00:07:21.402 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_server_request": 329, 2024-03-18T00:07:21.402 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_request": 7782, 2024-03-18T00:07:21.402 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_session": 8, 2024-03-18T00:07:21.402 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_peer_request": 3661, 2024-03-18T00:07:21.402 INFO:teuthology.orchestra.run.smithi191.stdout: "req_create_latency": { 2024-03-18T00:07:21.402 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1048, 2024-03-18T00:07:21.402 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 10.953540868, 2024-03-18T00:07:21.402 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.010451851 2024-03-18T00:07:21.402 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.402 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getattr_latency": { 2024-03-18T00:07:21.402 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.402 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getfilelock_latency": { 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: "req_link_latency": { 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookup_latency": { 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 3505, 2024-03-18T00:07:21.403 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 32.954963392, 2024-03-18T00:07:21.404 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.009402272 2024-03-18T00:07:21.404 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.404 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookuphash_latency": { 2024-03-18T00:07:21.404 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.404 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.404 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.404 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.404 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupino_latency": { 2024-03-18T00:07:21.404 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.404 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.404 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.404 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.405 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupname_latency": { 2024-03-18T00:07:21.405 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.405 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.405 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.405 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.405 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupparent_latency": { 2024-03-18T00:07:21.405 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.406 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.406 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.406 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupsnap_latency": { 2024-03-18T00:07:21.406 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.406 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.406 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.406 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lssnap_latency": { 2024-03-18T00:07:21.407 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.407 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.407 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.407 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mkdir_latency": { 2024-03-18T00:07:21.407 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.407 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.407 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.407 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mknod_latency": { 2024-03-18T00:07:21.407 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mksnap_latency": { 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: "req_open_latency": { 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 904, 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.597819037, 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000661304 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: "req_readdir_latency": { 2024-03-18T00:07:21.408 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 33, 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.045487672, 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001378414 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rename_latency": { 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1050, 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 12.887483127, 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.012273793 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: "req_renamesnap_latency": { 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.409 INFO:teuthology.orchestra.run.smithi191.stdout: "req_snapdiff_latency": { 2024-03-18T00:07:21.410 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.410 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.410 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.410 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmdir_latency": { 2024-03-18T00:07:21.410 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.410 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.410 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.410 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmsnap_latency": { 2024-03-18T00:07:21.410 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.410 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.410 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmxattr_latency": { 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setattr_latency": { 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setdirlayout_latency": { 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.411 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setfilelock_latency": { 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setlayout_latency": { 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setxattr_latency": { 2024-03-18T00:07:21.412 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: "req_symlink_latency": { 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: "req_unlink_latency": { 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 824, 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 1.945840000, 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.002361456 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_revoke_eviction": 0, 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_acquisition_throttle": 0, 2024-03-18T00:07:21.413 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getvxattr_latency": { 2024-03-18T00:07:21.414 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.414 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.414 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.414 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:21.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.414 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_sessions": { 2024-03-18T00:07:21.414 INFO:teuthology.orchestra.run.smithi191.stdout: "session_count": 1, 2024-03-18T00:07:21.414 INFO:teuthology.orchestra.run.smithi191.stdout: "session_add": 1, 2024-03-18T00:07:21.414 INFO:teuthology.orchestra.run.smithi191.stdout: "session_remove": 0, 2024-03-18T00:07:21.414 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_open": 1, 2024-03-18T00:07:21.414 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_stale": 0, 2024-03-18T00:07:21.414 INFO:teuthology.orchestra.run.smithi191.stdout: "total_load": 1228, 2024-03-18T00:07:21.414 INFO:teuthology.orchestra.run.smithi191.stdout: "average_load": 1228, 2024-03-18T00:07:21.415 INFO:teuthology.orchestra.run.smithi191.stdout: "avg_session_uptime": 505, 2024-03-18T00:07:21.415 INFO:teuthology.orchestra.run.smithi191.stdout: "mdthresh_evicted": 0 2024-03-18T00:07:21.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.415 INFO:teuthology.orchestra.run.smithi191.stdout: "mempool": { 2024-03-18T00:07:21.415 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_bytes": 0, 2024-03-18T00:07:21.415 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_items": 0, 2024-03-18T00:07:21.415 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T00:07:21.415 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_items": 0, 2024-03-18T00:07:21.415 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T00:07:21.415 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_items": 0, 2024-03-18T00:07:21.415 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T00:07:21.415 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T00:07:21.415 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_items": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_bytes": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_items": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_bytes": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_items": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T00:07:21.416 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_items": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_bytes": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_items": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_bytes": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_items": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_bytes": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_items": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_items": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_items": 0, 2024-03-18T00:07:21.417 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_bytes": 294495, 2024-03-18T00:07:21.418 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_items": 134, 2024-03-18T00:07:21.418 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_bytes": 704, 2024-03-18T00:07:21.418 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_items": 8, 2024-03-18T00:07:21.418 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_bytes": 0, 2024-03-18T00:07:21.418 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_items": 0, 2024-03-18T00:07:21.418 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_bytes": 0, 2024-03-18T00:07:21.418 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_items": 0, 2024-03-18T00:07:21.418 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_bytes": 0, 2024-03-18T00:07:21.418 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_items": 0, 2024-03-18T00:07:21.418 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_bytes": 8792, 2024-03-18T00:07:21.418 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_items": 168, 2024-03-18T00:07:21.419 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T00:07:21.419 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_items": 0, 2024-03-18T00:07:21.419 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_bytes": 0, 2024-03-18T00:07:21.419 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_items": 0, 2024-03-18T00:07:21.419 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_bytes": 131325, 2024-03-18T00:07:21.419 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_items": 546, 2024-03-18T00:07:21.419 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_bytes": 0, 2024-03-18T00:07:21.419 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_items": 0, 2024-03-18T00:07:21.419 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_bytes": 0, 2024-03-18T00:07:21.419 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_items": 0 2024-03-18T00:07:21.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: "objecter": { 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: "op_active": 0, 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: "op_laggy": 0, 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send": 3460, 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send_bytes": 11770527, 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: "op_resend": 33, 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: "op_reply": 3460, 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: "op_latency": { 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 3460, 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 64.699875750, 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.018699386 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: "op_inflight": 0, 2024-03-18T00:07:21.420 INFO:teuthology.orchestra.run.smithi191.stdout: "oplen_avg": { 2024-03-18T00:07:21.421 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 3460, 2024-03-18T00:07:21.421 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 3733 2024-03-18T00:07:21.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.421 INFO:teuthology.orchestra.run.smithi191.stdout: "op": 3460, 2024-03-18T00:07:21.421 INFO:teuthology.orchestra.run.smithi191.stdout: "op_r": 75, 2024-03-18T00:07:21.421 INFO:teuthology.orchestra.run.smithi191.stdout: "op_w": 3385, 2024-03-18T00:07:21.421 INFO:teuthology.orchestra.run.smithi191.stdout: "op_rmw": 0, 2024-03-18T00:07:21.421 INFO:teuthology.orchestra.run.smithi191.stdout: "op_pg": 0, 2024-03-18T00:07:21.421 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_stat": 0, 2024-03-18T00:07:21.421 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_create": 10, 2024-03-18T00:07:21.421 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_read": 75, 2024-03-18T00:07:21.421 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_write": 408, 2024-03-18T00:07:21.421 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writefull": 75, 2024-03-18T00:07:21.422 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writesame": 0, 2024-03-18T00:07:21.422 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_append": 0, 2024-03-18T00:07:21.422 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_zero": 0, 2024-03-18T00:07:21.422 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_truncate": 4, 2024-03-18T00:07:21.422 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_delete": 2664, 2024-03-18T00:07:21.422 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_mapext": 0, 2024-03-18T00:07:21.422 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_sparse_read": 0, 2024-03-18T00:07:21.422 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_clonerange": 0, 2024-03-18T00:07:21.422 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_getxattr": 0, 2024-03-18T00:07:21.422 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_setxattr": 20, 2024-03-18T00:07:21.422 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_cmpxattr": 0, 2024-03-18T00:07:21.422 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_rmxattr": 0, 2024-03-18T00:07:21.422 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_resetxattrs": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_call": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_watch": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_notify": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls_filter": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_other": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_active": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_send": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_resend": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_ping": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_active": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_send": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_resend": 0, 2024-03-18T00:07:21.423 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_active": 0, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_send": 0, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_resend": 0, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_active": 0, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_send": 0, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_resend": 0, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "command_active": 0, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "command_send": 0, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "command_resend": 0, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "map_epoch": 94, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "map_full": 0, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "map_inc": 4, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_sessions": 12, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_open": 12, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_close": 0, 2024-03-18T00:07:21.424 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_laggy": 0, 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_wr": 387, 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_rd": 0, 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_del": 90 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: "oft": { 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_objs": 1, 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_kv_pairs": 0, 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_updates": 0, 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_removes": 0 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: "osd-slow-ops": { 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_ops_count": 0 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.425 INFO:teuthology.orchestra.run.smithi191.stdout: "purge_queue": { 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops": 0, 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops_high_water": 10, 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing": 0, 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_high_water": 5, 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed_ops": 2650, 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed": 1325, 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_item_in_journal": 0 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 40216, 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 12635216, 2024-03-18T00:07:21.426 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 40216, 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 40216, 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 12635216, 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_bytes": { 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T00:07:21.427 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 3460, 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 11914952, 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 3460, 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 11914952, 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.428 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_ops": { 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 1024, 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 3460, 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 3460, 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 3460, 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 3460, 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.429 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:21.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.430 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle": { 2024-03-18T00:07:21.430 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:21.430 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T00:07:21.430 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:21.430 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 1325, 2024-03-18T00:07:21.430 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 144425, 2024-03-18T00:07:21.430 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:21.430 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 1325, 2024-03-18T00:07:21.430 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T00:07:21.430 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T00:07:21.430 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 75, 2024-03-18T00:07:21.431 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 144425, 2024-03-18T00:07:21.431 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:21.431 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.431 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.431 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.431 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:21.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:21.431 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle-0x555b59e54500": { 2024-03-18T00:07:21.431 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:21.431 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T00:07:21.431 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:21.431 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 7453, 2024-03-18T00:07:21.431 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 10762630, 2024-03-18T00:07:21.432 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:21.432 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 7453, 2024-03-18T00:07:21.432 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T00:07:21.432 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T00:07:21.432 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 333, 2024-03-18T00:07:21.432 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 10762630, 2024-03-18T00:07:21.432 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:21.432 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:21.432 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:21.432 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:21.432 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:21.432 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:21.432 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-18T00:07:21.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:21 smithi012 ceph-mon[31541]: pgmap v697: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T00:07:21.999 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:21 smithi191 ceph-mon[36708]: pgmap v697: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T00:07:22.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:21 smithi079 ceph-mon[36241]: pgmap v697: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T00:07:22.213 INFO:tasks.check_counter:Daemon mds.i mds.exported=33 2024-03-18T00:07:22.214 INFO:tasks.check_counter:Daemon mds.i mds.imported=5 2024-03-18T00:07:22.214 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=436 2024-03-18T00:07:22.214 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=4393 2024-03-18T00:07:22.214 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=4 2024-03-18T00:07:22.214 DEBUG:tasks.check_counter:Getting stats from l 2024-03-18T00:07:22.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-18T00:07:22.466 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:07:23.897 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:23 smithi191 ceph-mon[36708]: pgmap v698: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-03-18T00:07:23.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:23 smithi012 ceph-mon[31541]: pgmap v698: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-03-18T00:07:24.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:23 smithi079 ceph-mon[36241]: pgmap v698: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-03-18T00:07:24.299 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-18T00:07:24.299 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T00:07:24.299 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 679, 2024-03-18T00:07:24.299 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 679, 2024-03-18T00:07:24.299 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 229817, 2024-03-18T00:07:24.299 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 215099, 2024-03-18T00:07:24.299 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 6, 2024-03-18T00:07:24.299 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 2, 2024-03-18T00:07:24.299 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.052106205, 2024-03-18T00:07:24.299 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.015471259, 2024-03-18T00:07:24.300 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.014204240, 2024-03-18T00:07:24.300 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.017163991, 2024-03-18T00:07:24.300 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:24.300 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 678, 2024-03-18T00:07:24.300 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.025155507, 2024-03-18T00:07:24.300 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000037102 2024-03-18T00:07:24.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.300 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:24.300 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.300 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.300 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.300 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-18T00:07:24.301 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T00:07:24.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.301 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T00:07:24.301 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 1371, 2024-03-18T00:07:24.301 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 1346, 2024-03-18T00:07:24.301 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 4594932, 2024-03-18T00:07:24.301 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 489807, 2024-03-18T00:07:24.301 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 4, 2024-03-18T00:07:24.301 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 3, 2024-03-18T00:07:24.301 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.095067725, 2024-03-18T00:07:24.301 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.027600277, 2024-03-18T00:07:24.302 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.040541962, 2024-03-18T00:07:24.302 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.026990387, 2024-03-18T00:07:24.302 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:24.302 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1345, 2024-03-18T00:07:24.302 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.035716383, 2024-03-18T00:07:24.302 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000026554 2024-03-18T00:07:24.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.302 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:24.302 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.302 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.302 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.303 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 903984, 2024-03-18T00:07:24.303 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 132000 2024-03-18T00:07:24.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.303 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T00:07:24.303 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 4, 2024-03-18T00:07:24.303 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 165, 2024-03-18T00:07:24.303 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 1508, 2024-03-18T00:07:24.303 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 349221, 2024-03-18T00:07:24.303 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 5, 2024-03-18T00:07:24.303 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 4, 2024-03-18T00:07:24.303 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.007555538, 2024-03-18T00:07:24.304 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.005500069, 2024-03-18T00:07:24.304 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.000860799, 2024-03-18T00:07:24.304 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.000062938, 2024-03-18T00:07:24.304 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T00:07:24.304 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 163, 2024-03-18T00:07:24.304 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.008070548, 2024-03-18T00:07:24.304 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000049512 2024-03-18T00:07:24.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.304 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T00:07:24.304 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.304 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.304 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.305 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T00:07:24.305 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 348256 2024-03-18T00:07:24.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.305 INFO:teuthology.orchestra.run.smithi191.stdout: "cct": { 2024-03-18T00:07:24.305 INFO:teuthology.orchestra.run.smithi191.stdout: "total_workers": 1, 2024-03-18T00:07:24.305 INFO:teuthology.orchestra.run.smithi191.stdout: "unhealthy_workers": 0 2024-03-18T00:07:24.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.305 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-MDSRank": { 2024-03-18T00:07:24.305 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T00:07:24.305 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T00:07:24.305 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 3215, 2024-03-18T00:07:24.305 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.066002181, 2024-03-18T00:07:24.305 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000020529 2024-03-18T00:07:24.306 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:24.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.306 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-PurgeQueue": { 2024-03-18T00:07:24.306 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T00:07:24.306 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T00:07:24.306 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.306 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.306 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.306 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:24.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.306 INFO:teuthology.orchestra.run.smithi191.stdout: "mds": { 2024-03-18T00:07:24.306 INFO:teuthology.orchestra.run.smithi191.stdout: "request": 0, 2024-03-18T00:07:24.307 INFO:teuthology.orchestra.run.smithi191.stdout: "reply": 0, 2024-03-18T00:07:24.307 INFO:teuthology.orchestra.run.smithi191.stdout: "reply_latency": { 2024-03-18T00:07:24.307 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.307 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.307 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.307 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_reply": 0, 2024-03-18T00:07:24.307 INFO:teuthology.orchestra.run.smithi191.stdout: "forward": 0, 2024-03-18T00:07:24.307 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_complete": 0, 2024-03-18T00:07:24.307 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_keys": 0, 2024-03-18T00:07:24.307 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_commit": 0, 2024-03-18T00:07:24.307 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_split": 0, 2024-03-18T00:07:24.307 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_merge": 0, 2024-03-18T00:07:24.308 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes": 415, 2024-03-18T00:07:24.308 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_top": 0, 2024-03-18T00:07:24.308 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_bottom": 415, 2024-03-18T00:07:24.308 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pin_tail": 0, 2024-03-18T00:07:24.308 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pinned": 415, 2024-03-18T00:07:24.308 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_expired": 0, 2024-03-18T00:07:24.308 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_with_caps": 0, 2024-03-18T00:07:24.308 INFO:teuthology.orchestra.run.smithi191.stdout: "caps": 0, 2024-03-18T00:07:24.308 INFO:teuthology.orchestra.run.smithi191.stdout: "subtrees": 1, 2024-03-18T00:07:24.308 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse": 0, 2024-03-18T00:07:24.308 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_hit": 0, 2024-03-18T00:07:24.308 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_forward": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_discover": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_dir_fetch": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_remote_ino": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_lock": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "load_cent": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "q": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "exported": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "exported_inodes": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "imported": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "imported_inodes": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_dir_fetch": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_backtrace_fetch": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_peer_discover": 0, 2024-03-18T00:07:24.309 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rfiles": 0, 2024-03-18T00:07:24.310 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rbytes": 0, 2024-03-18T00:07:24.310 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rsnaps": 0, 2024-03-18T00:07:24.310 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T00:07:24.310 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_set_tag": 0, 2024-03-18T00:07:24.310 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T00:07:24.310 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_inotable_repaired": 0, 2024-03-18T00:07:24.310 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_inodes": 0, 2024-03-18T00:07:24.310 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T00:07:24.310 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T00:07:24.310 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_file_inodes": 0, 2024-03-18T00:07:24.310 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_inode_file_caps": 0, 2024-03-18T00:07:24.310 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_grant": 0, 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps": 0, 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps_dirty": 0, 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_cap_release": 0, 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: "process_request_cap_release": 0 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_cache": { 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update": 0, 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update_receipt": 0, 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_try_discover": 0, 2024-03-18T00:07:24.311 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_send_discover": 0, 2024-03-18T00:07:24.312 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_handle_discover": 0, 2024-03-18T00:07:24.312 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays": 0, 2024-03-18T00:07:24.312 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_delayed": 0, 2024-03-18T00:07:24.312 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_enqueuing": 0, 2024-03-18T00:07:24.312 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_created": 0, 2024-03-18T00:07:24.312 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_enqueued": 0, 2024-03-18T00:07:24.312 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_reintegrated": 0, 2024-03-18T00:07:24.312 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_migrated": 0, 2024-03-18T00:07:24.312 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_processing": 0, 2024-03-18T00:07:24.312 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_enqueued": 0, 2024-03-18T00:07:24.312 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_prioritized": 0, 2024-03-18T00:07:24.313 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_started": 0, 2024-03-18T00:07:24.313 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_completed": 0, 2024-03-18T00:07:24.313 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_path": 0, 2024-03-18T00:07:24.313 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_inode": 0, 2024-03-18T00:07:24.313 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T00:07:24.313 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_exportdir": 0, 2024-03-18T00:07:24.313 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_flush": 0, 2024-03-18T00:07:24.313 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragmentdir": 0, 2024-03-18T00:07:24.313 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragstats": 0, 2024-03-18T00:07:24.313 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_inodestats": 0 2024-03-18T00:07:24.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.313 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_log": { 2024-03-18T00:07:24.313 INFO:teuthology.orchestra.run.smithi191.stdout: "evlrg": 0, 2024-03-18T00:07:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: "evadd": 0, 2024-03-18T00:07:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: "evex": 0, 2024-03-18T00:07:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: "evtrm": 0, 2024-03-18T00:07:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: "ev": 2385, 2024-03-18T00:07:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: "evexg": 0, 2024-03-18T00:07:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: "evexd": 0, 2024-03-18T00:07:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: "segadd": 0, 2024-03-18T00:07:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: "segex": 0, 2024-03-18T00:07:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: "segtrm": 0, 2024-03-18T00:07:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: "seg": 4, 2024-03-18T00:07:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: "segmjr": 2, 2024-03-18T00:07:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: "segexg": 0, 2024-03-18T00:07:24.315 INFO:teuthology.orchestra.run.smithi191.stdout: "segexd": 0, 2024-03-18T00:07:24.315 INFO:teuthology.orchestra.run.smithi191.stdout: "expos": 4194304, 2024-03-18T00:07:24.315 INFO:teuthology.orchestra.run.smithi191.stdout: "wrpos": 7597654, 2024-03-18T00:07:24.315 INFO:teuthology.orchestra.run.smithi191.stdout: "rdpos": 7596516, 2024-03-18T00:07:24.315 INFO:teuthology.orchestra.run.smithi191.stdout: "jlat": { 2024-03-18T00:07:24.315 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.315 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.315 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.315 INFO:teuthology.orchestra.run.smithi191.stdout: "replayed": 2385 2024-03-18T00:07:24.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.315 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_mem": { 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "ino": 6, 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "ino+": 433, 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "ino-": 427, 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "dir": 4, 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "dir+": 45, 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "dir-": 41, 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "dn": 415, 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "dn+": 1158, 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "dn-": 743, 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "cap": 0, 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "cap+": 0, 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "cap-": 0, 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "rss": 49060, 2024-03-18T00:07:24.316 INFO:teuthology.orchestra.run.smithi191.stdout: "heap": 215420 2024-03-18T00:07:24.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.317 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_server": { 2024-03-18T00:07:24.317 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_client_request": 0, 2024-03-18T00:07:24.317 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_server_request": 0, 2024-03-18T00:07:24.317 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_request": 0, 2024-03-18T00:07:24.317 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_session": 0, 2024-03-18T00:07:24.317 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_peer_request": 0, 2024-03-18T00:07:24.317 INFO:teuthology.orchestra.run.smithi191.stdout: "req_create_latency": { 2024-03-18T00:07:24.317 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.317 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.317 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.317 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getattr_latency": { 2024-03-18T00:07:24.318 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.318 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.318 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.318 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getfilelock_latency": { 2024-03-18T00:07:24.318 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.318 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.318 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.318 INFO:teuthology.orchestra.run.smithi191.stdout: "req_link_latency": { 2024-03-18T00:07:24.318 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.318 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.318 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookup_latency": { 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookuphash_latency": { 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupino_latency": { 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.319 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupname_latency": { 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupparent_latency": { 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupsnap_latency": { 2024-03-18T00:07:24.320 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.321 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.321 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.321 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lssnap_latency": { 2024-03-18T00:07:24.321 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.321 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.321 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.321 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mkdir_latency": { 2024-03-18T00:07:24.322 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.322 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.322 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.322 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mknod_latency": { 2024-03-18T00:07:24.322 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.322 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mksnap_latency": { 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: "req_open_latency": { 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: "req_readdir_latency": { 2024-03-18T00:07:24.323 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.324 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.324 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.324 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rename_latency": { 2024-03-18T00:07:24.324 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.324 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.324 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.324 INFO:teuthology.orchestra.run.smithi191.stdout: "req_renamesnap_latency": { 2024-03-18T00:07:24.324 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.324 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.324 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.325 INFO:teuthology.orchestra.run.smithi191.stdout: "req_snapdiff_latency": { 2024-03-18T00:07:24.325 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.325 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.325 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.325 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmdir_latency": { 2024-03-18T00:07:24.325 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.325 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.325 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.325 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmsnap_latency": { 2024-03-18T00:07:24.325 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.326 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.326 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.326 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmxattr_latency": { 2024-03-18T00:07:24.326 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.326 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.326 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.326 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setattr_latency": { 2024-03-18T00:07:24.326 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.326 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.326 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setdirlayout_latency": { 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setfilelock_latency": { 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setlayout_latency": { 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.327 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setxattr_latency": { 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: "req_symlink_latency": { 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: "req_unlink_latency": { 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.328 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.329 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.329 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_revoke_eviction": 0, 2024-03-18T00:07:24.329 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_acquisition_throttle": 0, 2024-03-18T00:07:24.329 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getvxattr_latency": { 2024-03-18T00:07:24.329 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.329 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.329 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.329 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:24.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.329 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_sessions": { 2024-03-18T00:07:24.329 INFO:teuthology.orchestra.run.smithi191.stdout: "session_count": 1, 2024-03-18T00:07:24.329 INFO:teuthology.orchestra.run.smithi191.stdout: "session_add": 1, 2024-03-18T00:07:24.330 INFO:teuthology.orchestra.run.smithi191.stdout: "session_remove": 0, 2024-03-18T00:07:24.330 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_open": 1, 2024-03-18T00:07:24.330 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_stale": 0, 2024-03-18T00:07:24.330 INFO:teuthology.orchestra.run.smithi191.stdout: "total_load": 0, 2024-03-18T00:07:24.330 INFO:teuthology.orchestra.run.smithi191.stdout: "average_load": 0, 2024-03-18T00:07:24.330 INFO:teuthology.orchestra.run.smithi191.stdout: "avg_session_uptime": 2587, 2024-03-18T00:07:24.330 INFO:teuthology.orchestra.run.smithi191.stdout: "mdthresh_evicted": 0 2024-03-18T00:07:24.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.330 INFO:teuthology.orchestra.run.smithi191.stdout: "mempool": { 2024-03-18T00:07:24.330 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_bytes": 0, 2024-03-18T00:07:24.330 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_items": 0, 2024-03-18T00:07:24.330 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T00:07:24.331 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_items": 0, 2024-03-18T00:07:24.331 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T00:07:24.331 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_items": 0, 2024-03-18T00:07:24.331 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T00:07:24.331 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T00:07:24.331 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T00:07:24.331 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T00:07:24.331 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T00:07:24.331 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_items": 0, 2024-03-18T00:07:24.331 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T00:07:24.331 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T00:07:24.331 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_bytes": 0, 2024-03-18T00:07:24.331 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_items": 0, 2024-03-18T00:07:24.332 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_bytes": 0, 2024-03-18T00:07:24.332 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_items": 0, 2024-03-18T00:07:24.332 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T00:07:24.332 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T00:07:24.332 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T00:07:24.332 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T00:07:24.332 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T00:07:24.332 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_items": 0, 2024-03-18T00:07:24.332 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_bytes": 0, 2024-03-18T00:07:24.332 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_items": 0, 2024-03-18T00:07:24.332 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T00:07:24.332 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T00:07:24.332 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_bytes": 0, 2024-03-18T00:07:24.333 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_items": 0, 2024-03-18T00:07:24.333 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_bytes": 0, 2024-03-18T00:07:24.333 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_items": 0, 2024-03-18T00:07:24.333 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T00:07:24.333 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_items": 0, 2024-03-18T00:07:24.333 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T00:07:24.333 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_items": 0, 2024-03-18T00:07:24.333 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_bytes": 197441, 2024-03-18T00:07:24.333 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_items": 54, 2024-03-18T00:07:24.333 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_bytes": 0, 2024-03-18T00:07:24.333 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_items": 0, 2024-03-18T00:07:24.333 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_bytes": 0, 2024-03-18T00:07:24.334 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_items": 0, 2024-03-18T00:07:24.334 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_bytes": 0, 2024-03-18T00:07:24.334 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_items": 0, 2024-03-18T00:07:24.334 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_bytes": 0, 2024-03-18T00:07:24.334 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_items": 0, 2024-03-18T00:07:24.334 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_bytes": 8792, 2024-03-18T00:07:24.334 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_items": 168, 2024-03-18T00:07:24.334 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T00:07:24.334 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_items": 0, 2024-03-18T00:07:24.334 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_bytes": 0, 2024-03-18T00:07:24.334 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_items": 0, 2024-03-18T00:07:24.334 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_bytes": 311128, 2024-03-18T00:07:24.334 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_items": 1711, 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_bytes": 0, 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_items": 0, 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_bytes": 0, 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_items": 0 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: "objecter": { 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: "op_active": 0, 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: "op_laggy": 0, 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send": 1803, 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send_bytes": 21, 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: "op_resend": 3, 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: "op_reply": 1803, 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: "op_latency": { 2024-03-18T00:07:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1803, 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 42.696256418, 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.023680674 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: "op_inflight": 0, 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: "oplen_avg": { 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1803, 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 1804 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: "op": 1803, 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: "op_r": 1803, 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: "op_w": 0, 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: "op_rmw": 0, 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: "op_pg": 0, 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_stat": 1056, 2024-03-18T00:07:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_create": 0, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_read": 745, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_write": 0, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writefull": 0, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writesame": 0, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_append": 0, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_zero": 0, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_truncate": 0, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_delete": 0, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_mapext": 0, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_sparse_read": 0, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_clonerange": 0, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_getxattr": 1, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_setxattr": 0, 2024-03-18T00:07:24.337 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_cmpxattr": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_rmxattr": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_resetxattrs": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_call": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_watch": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_notify": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls_filter": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_other": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_active": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_send": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_resend": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_ping": 0, 2024-03-18T00:07:24.338 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_active": 0, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_send": 0, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_resend": 0, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_active": 0, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_send": 0, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_resend": 0, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_active": 0, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_send": 0, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_resend": 0, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "command_active": 0, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "command_send": 0, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "command_resend": 0, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "map_epoch": 94, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "map_full": 0, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "map_inc": 4, 2024-03-18T00:07:24.339 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_sessions": 6, 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_open": 6, 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_close": 0, 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_laggy": 0, 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_wr": 0, 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_rd": 2, 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_del": 0 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: "oft": { 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_objs": 0, 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_kv_pairs": 0, 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_updates": 0, 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_removes": 0 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.340 INFO:teuthology.orchestra.run.smithi191.stdout: "osd-slow-ops": { 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_ops_count": 0 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: "purge_queue": { 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops": 0, 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing": 0, 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_high_water": 0, 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed_ops": 0, 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed": 0, 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_item_in_journal": 0 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:24.341 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T00:07:24.342 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:24.342 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 2050, 2024-03-18T00:07:24.342 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 4626161, 2024-03-18T00:07:24.342 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:24.342 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 2050, 2024-03-18T00:07:24.342 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T00:07:24.342 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T00:07:24.342 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 2050, 2024-03-18T00:07:24.342 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 4626161, 2024-03-18T00:07:24.342 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:24.342 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.342 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_bytes": { 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 1803, 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 3403355, 2024-03-18T00:07:24.343 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 1803, 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 3403355, 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_ops": { 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 1024, 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T00:07:24.344 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 1803, 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 1803, 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 1803, 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 1803, 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle": { 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:24.345 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 0, 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 0, 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.346 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle-0x558ea88086e0": { 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 0, 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 0, 2024-03-18T00:07:24.347 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T00:07:24.348 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T00:07:24.348 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T00:07:24.348 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T00:07:24.348 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:24.348 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T00:07:24.348 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-18T00:07:25.081 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-18T00:07:25.081 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-18T00:07:25.081 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-03-18T00:07:25.081 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-03-18T00:07:25.081 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-03-18T00:07:25.081 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-18T00:07:25.090 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-18T00:07:25.091 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-18T00:07:25.091 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T00:07:25.149 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-18T00:07:25.149 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T00:07:25.204 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:07:25.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.420+0000 7f9682600640 1 -- 172.21.15.12:0/3950913007 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f967c075820 msgr2=0x7f967c073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:25.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.420+0000 7f9682600640 1 --2- 172.21.15.12:0/3950913007 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f967c075820 0x7f967c073c70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9670009930 tx=0x7f967002bc30 comp rx=0 tx=0).stop 2024-03-18T00:07:25.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.420+0000 7f9682600640 1 -- 172.21.15.12:0/3950913007 shutdown_connections 2024-03-18T00:07:25.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.420+0000 7f9682600640 1 --2- 172.21.15.12:0/3950913007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f967c10dbc0 0x7f967c10ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.420+0000 7f9682600640 1 --2- 172.21.15.12:0/3950913007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f967c074240 0x7f967c0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.420+0000 7f9682600640 1 --2- 172.21.15.12:0/3950913007 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f967c075820 0x7f967c073c70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.420+0000 7f9682600640 1 -- 172.21.15.12:0/3950913007 >> 172.21.15.12:0/3950913007 conn(0x7f967c0fcb40 msgr2=0x7f967c0fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:25.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.420+0000 7f9682600640 1 -- 172.21.15.12:0/3950913007 shutdown_connections 2024-03-18T00:07:25.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.420+0000 7f9682600640 1 -- 172.21.15.12:0/3950913007 wait complete. 2024-03-18T00:07:25.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.421+0000 7f9682600640 1 Processor -- start 2024-03-18T00:07:25.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.421+0000 7f9682600640 1 -- start start 2024-03-18T00:07:25.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.421+0000 7f9682600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f967c074240 0x7f967c19ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:25.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.421+0000 7f9682600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f967c075820 0x7f967c19e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:25.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.421+0000 7f9682600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f967c10dbc0 0x7f967c1a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:25.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.421+0000 7f9682600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f967c113890 con 0x7f967c075820 2024-03-18T00:07:25.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.421+0000 7f9682600640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f967c113710 con 0x7f967c10dbc0 2024-03-18T00:07:25.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.421+0000 7f9682600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f967c113a10 con 0x7f967c074240 2024-03-18T00:07:25.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.422+0000 7f9681200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f967c074240 0x7f967c19ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:25.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.422+0000 7f9681200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f967c074240 0x7f967c19ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:39868/0 (socket says 172.21.15.12:39868) 2024-03-18T00:07:25.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.422+0000 7f9681200640 1 -- 172.21.15.12:0/3380010403 learned_addr learned my addr 172.21.15.12:0/3380010403 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:25.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.422+0000 7f9681c00640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f967c10dbc0 0x7f967c1a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:25.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.422+0000 7f967be00640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f967c075820 0x7f967c19e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:25.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.422+0000 7f9681200640 1 -- 172.21.15.12:0/3380010403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f967c10dbc0 msgr2=0x7f967c1a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:25.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.422+0000 7f9681200640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f967c10dbc0 0x7f967c1a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.422+0000 7f9681200640 1 -- 172.21.15.12:0/3380010403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f967c075820 msgr2=0x7f967c19e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:25.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.422+0000 7f9681200640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f967c075820 0x7f967c19e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.422+0000 7f9681200640 1 -- 172.21.15.12:0/3380010403 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967c1a5af0 con 0x7f967c074240 2024-03-18T00:07:25.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.422+0000 7f967be00640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f967c075820 0x7f967c19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:07:25.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.423+0000 7f9681200640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f967c074240 0x7f967c19ded0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9670000c00 tx=0x7f967002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:25.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.423+0000 7f9679600640 1 -- 172.21.15.12:0/3380010403 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9670043020 con 0x7f967c074240 2024-03-18T00:07:25.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.423+0000 7f9682600640 1 -- 172.21.15.12:0/3380010403 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f967c1a5d80 con 0x7f967c074240 2024-03-18T00:07:25.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.423+0000 7f9679600640 1 -- 172.21.15.12:0/3380010403 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96700054e0 con 0x7f967c074240 2024-03-18T00:07:25.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.423+0000 7f9679600640 1 -- 172.21.15.12:0/3380010403 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9670003c40 con 0x7f967c074240 2024-03-18T00:07:25.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.423+0000 7f9682600640 1 -- 172.21.15.12:0/3380010403 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f967c1a6240 con 0x7f967c074240 2024-03-18T00:07:25.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.424+0000 7f9682600640 1 -- 172.21.15.12:0/3380010403 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f967c103d90 con 0x7f967c074240 2024-03-18T00:07:25.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.426+0000 7f9679600640 1 -- 172.21.15.12:0/3380010403 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f9670046020 con 0x7f967c074240 2024-03-18T00:07:25.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.427+0000 7f9679600640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f96640a2880 0x7f96640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:25.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.428+0000 7f9679600640 1 -- 172.21.15.12:0/3380010403 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f96700f0e50 con 0x7f967c074240 2024-03-18T00:07:25.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.428+0000 7f967be00640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f96640a2880 0x7f96640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:25.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.429+0000 7f967be00640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f96640a2880 0x7f96640a4d40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f967c19f3e0 tx=0x7f9668005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:25.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.436+0000 7f9679600640 1 -- 172.21.15.12:0/3380010403 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f967004d020 con 0x7f967c074240 2024-03-18T00:07:25.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.648+0000 7f9682600640 1 -- 172.21.15.12:0/3380010403 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f967c062560 con 0x7f967c074240 2024-03-18T00:07:25.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.650+0000 7f9679600640 1 -- 172.21.15.12:0/3380010403 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24618 (secure 0 0 0) 0x7f96700bdb10 con 0x7f967c074240 2024-03-18T00:07:25.651 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-18T00:07:25.651 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":94,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:56:55.719277+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:07:25.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.653+0000 7f9682600640 1 -- 172.21.15.12:0/3380010403 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f96640a2880 msgr2=0x7f96640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:25.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.653+0000 7f9682600640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f96640a2880 0x7f96640a4d40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f967c19f3e0 tx=0x7f9668005f90 comp rx=0 tx=0).stop 2024-03-18T00:07:25.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.653+0000 7f9682600640 1 -- 172.21.15.12:0/3380010403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f967c074240 msgr2=0x7f967c19ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:25.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.653+0000 7f9682600640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f967c074240 0x7f967c19ded0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9670000c00 tx=0x7f967002c650 comp rx=0 tx=0).stop 2024-03-18T00:07:25.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.653+0000 7f9682600640 1 -- 172.21.15.12:0/3380010403 shutdown_connections 2024-03-18T00:07:25.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.653+0000 7f9682600640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f96640a2880 0x7f96640a4d40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.653+0000 7f9682600640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f967c10dbc0 0x7f967c1a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.653+0000 7f9682600640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f967c075820 0x7f967c19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.653+0000 7f9682600640 1 --2- 172.21.15.12:0/3380010403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f967c074240 0x7f967c19ded0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.653+0000 7f9682600640 1 -- 172.21.15.12:0/3380010403 >> 172.21.15.12:0/3380010403 conn(0x7f967c0fcb40 msgr2=0x7f967c100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:25.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.654+0000 7f9682600640 1 -- 172.21.15.12:0/3380010403 shutdown_connections 2024-03-18T00:07:25.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.654+0000 7f9682600640 1 -- 172.21.15.12:0/3380010403 wait complete. 2024-03-18T00:07:25.673 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:49:38.640282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:54:54.540345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:54:55.803926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:55:00.355066+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T00:07:25.673 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T00:07:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.893+0000 7fb023e00640 1 -- 172.21.15.12:0/184381794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb024100890 msgr2=0x7fb024100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.893+0000 7fb023e00640 1 --2- 172.21.15.12:0/184381794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb024100890 0x7fb024100cf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009900 tx=0x7fb00c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:07:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.893+0000 7fb023e00640 1 -- 172.21.15.12:0/184381794 shutdown_connections 2024-03-18T00:07:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.893+0000 7fb023e00640 1 --2- 172.21.15.12:0/184381794 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb024101230 0x7fb0241016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.893+0000 7fb023e00640 1 --2- 172.21.15.12:0/184381794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb024100890 0x7fb024100cf0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.893+0000 7fb023e00640 1 --2- 172.21.15.12:0/184381794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0240fff30 0x7fb024100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.893+0000 7fb023e00640 1 -- 172.21.15.12:0/184381794 >> 172.21.15.12:0/184381794 conn(0x7fb0240fbb50 msgr2=0x7fb0240fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.894+0000 7fb023e00640 1 -- 172.21.15.12:0/184381794 shutdown_connections 2024-03-18T00:07:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.894+0000 7fb023e00640 1 -- 172.21.15.12:0/184381794 wait complete. 2024-03-18T00:07:25.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.895+0000 7fb023e00640 1 Processor -- start 2024-03-18T00:07:25.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.895+0000 7fb023e00640 1 -- start start 2024-03-18T00:07:25.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.895+0000 7fb023e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0240fff30 0x7fb024199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:25.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.895+0000 7fb023e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb024100890 0x7fb02419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:25.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb023e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb024101230 0x7fb0241a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:25.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb023e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb02410f580 con 0x7fb0240fff30 2024-03-18T00:07:25.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb023e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb02410f400 con 0x7fb024101230 2024-03-18T00:07:25.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb023e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb02410f700 con 0x7fb024100890 2024-03-18T00:07:25.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb022a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0240fff30 0x7fb024199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:25.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb01be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb024100890 0x7fb02419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:25.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb023400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb024101230 0x7fb0241a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:25.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb01be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb024100890 0x7fb02419a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:39888/0 (socket says 172.21.15.12:39888) 2024-03-18T00:07:25.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb022a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0240fff30 0x7fb024199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46942/0 (socket says 172.21.15.12:46942) 2024-03-18T00:07:25.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb01be00640 1 -- 172.21.15.12:0/2346137895 learned_addr learned my addr 172.21.15.12:0/2346137895 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:25.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb01be00640 1 -- 172.21.15.12:0/2346137895 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb024101230 msgr2=0x7fb0241a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:25.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb01be00640 1 --2- 172.21.15.12:0/2346137895 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb024101230 0x7fb0241a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb01be00640 1 -- 172.21.15.12:0/2346137895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0240fff30 msgr2=0x7fb024199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:25.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb01be00640 1 --2- 172.21.15.12:0/2346137895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0240fff30 0x7fb024199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:25.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb01be00640 1 -- 172.21.15.12:0/2346137895 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0241a17b0 con 0x7fb024100890 2024-03-18T00:07:25.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.896+0000 7fb023400640 1 --2- 172.21.15.12:0/2346137895 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb024101230 0x7fb0241a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:07:25.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.897+0000 7fb01be00640 1 --2- 172.21.15.12:0/2346137895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb024100890 0x7fb02419a0d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb00c000c00 tx=0x7fb00c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.897+0000 7fb01b400640 1 -- 172.21.15.12:0/2346137895 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c043020 con 0x7fb024100890 2024-03-18T00:07:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.897+0000 7fb01b400640 1 -- 172.21.15.12:0/2346137895 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb00c0054e0 con 0x7fb024100890 2024-03-18T00:07:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.897+0000 7fb01b400640 1 -- 172.21.15.12:0/2346137895 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c003cd0 con 0x7fb024100890 2024-03-18T00:07:25.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.897+0000 7fb023e00640 1 -- 172.21.15.12:0/2346137895 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0241a1a40 con 0x7fb024100890 2024-03-18T00:07:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.897+0000 7fb023e00640 1 -- 172.21.15.12:0/2346137895 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0241a1f40 con 0x7fb024100890 2024-03-18T00:07:25.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.898+0000 7fb023e00640 1 -- 172.21.15.12:0/2346137895 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb02410adc0 con 0x7fb024100890 2024-03-18T00:07:25.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.901+0000 7fb01b400640 1 -- 172.21.15.12:0/2346137895 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fb00c046020 con 0x7fb024100890 2024-03-18T00:07:25.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.902+0000 7fb01b400640 1 --2- 172.21.15.12:0/2346137895 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb0080a2880 0x7fb0080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:25.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.902+0000 7fb01b400640 1 -- 172.21.15.12:0/2346137895 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fb00c0f0d50 con 0x7fb024100890 2024-03-18T00:07:25.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.903+0000 7fb022a00640 1 --2- 172.21.15.12:0/2346137895 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb0080a2880 0x7fb0080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:25.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.903+0000 7fb022a00640 1 --2- 172.21.15.12:0/2346137895 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb0080a2880 0x7fb0080a4d40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb014009bd0 tx=0x7fb014005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:25.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:25.910+0000 7fb01b400640 1 -- 172.21.15.12:0/2346137895 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb00c0f4030 con 0x7fb024100890 2024-03-18T00:07:26.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:25 smithi079 ceph-mon[36241]: pgmap v699: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T00:07:26.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:25 smithi079 ceph-mon[36241]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:07:26.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:25 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3380010403' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:07:26.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:25 smithi191 ceph-mon[36708]: pgmap v699: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T00:07:26.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:25 smithi191 ceph-mon[36708]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:07:26.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:25 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3380010403' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:07:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.121+0000 7fb023e00640 1 -- 172.21.15.12:0/2346137895 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb0240623e0 con 0x7fb024100890 2024-03-18T00:07:26.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.122+0000 7fb01b400640 1 -- 172.21.15.12:0/2346137895 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fb00c04d020 con 0x7fb024100890 2024-03-18T00:07:26.123 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-18T00:07:26.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.125+0000 7fb023e00640 1 -- 172.21.15.12:0/2346137895 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb0080a2880 msgr2=0x7fb0080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.125+0000 7fb023e00640 1 --2- 172.21.15.12:0/2346137895 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb0080a2880 0x7fb0080a4d40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb014009bd0 tx=0x7fb014005f90 comp rx=0 tx=0).stop 2024-03-18T00:07:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.125+0000 7fb023e00640 1 -- 172.21.15.12:0/2346137895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb024100890 msgr2=0x7fb02419a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.125+0000 7fb023e00640 1 --2- 172.21.15.12:0/2346137895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb024100890 0x7fb02419a0d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb00c000c00 tx=0x7fb00c02c650 comp rx=0 tx=0).stop 2024-03-18T00:07:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.125+0000 7fb023e00640 1 -- 172.21.15.12:0/2346137895 shutdown_connections 2024-03-18T00:07:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.125+0000 7fb023e00640 1 --2- 172.21.15.12:0/2346137895 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fb0080a2880 0x7fb0080a4d40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.125+0000 7fb023e00640 1 --2- 172.21.15.12:0/2346137895 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb024101230 0x7fb0241a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.125+0000 7fb023e00640 1 --2- 172.21.15.12:0/2346137895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb024100890 0x7fb02419a0d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.125+0000 7fb023e00640 1 --2- 172.21.15.12:0/2346137895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0240fff30 0x7fb024199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.125+0000 7fb023e00640 1 -- 172.21.15.12:0/2346137895 >> 172.21.15.12:0/2346137895 conn(0x7fb0240fbb50 msgr2=0x7fb0240fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.125+0000 7fb023e00640 1 -- 172.21.15.12:0/2346137895 shutdown_connections 2024-03-18T00:07:26.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.125+0000 7fb023e00640 1 -- 172.21.15.12:0/2346137895 wait complete. 2024-03-18T00:07:26.144 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:25 smithi012 ceph-mon[31541]: pgmap v699: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T00:07:26.144 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:25 smithi012 ceph-mon[31541]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:07:26.144 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:25 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3380010403' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:07:26.146 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T00:07:26.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.381+0000 7fdac0a00640 1 -- 172.21.15.12:0/3489655436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdabc103290 msgr2=0x7fdabc109b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.381+0000 7fdac0a00640 1 --2- 172.21.15.12:0/3489655436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdabc103290 0x7fdabc109b20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009ca0 tx=0x7fdab402c100 comp rx=0 tx=0).stop 2024-03-18T00:07:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.381+0000 7fdac0a00640 1 -- 172.21.15.12:0/3489655436 shutdown_connections 2024-03-18T00:07:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.381+0000 7fdac0a00640 1 --2- 172.21.15.12:0/3489655436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdabc103290 0x7fdabc109b20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.381+0000 7fdac0a00640 1 --2- 172.21.15.12:0/3489655436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdabc1028d0 0x7fdabc102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.381+0000 7fdac0a00640 1 --2- 172.21.15.12:0/3489655436 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdabc1016e0 0x7fdabc101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.381+0000 7fdac0a00640 1 -- 172.21.15.12:0/3489655436 >> 172.21.15.12:0/3489655436 conn(0x7fdabc0fcc50 msgr2=0x7fdabc0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.381+0000 7fdac0a00640 1 -- 172.21.15.12:0/3489655436 shutdown_connections 2024-03-18T00:07:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.381+0000 7fdac0a00640 1 -- 172.21.15.12:0/3489655436 wait complete. 2024-03-18T00:07:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.382+0000 7fdac0a00640 1 Processor -- start 2024-03-18T00:07:26.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.383+0000 7fdac0a00640 1 -- start start 2024-03-18T00:07:26.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.383+0000 7fdac0a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdabc1016e0 0x7fdabc199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:26.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.383+0000 7fdac0a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdabc1028d0 0x7fdabc19a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:26.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.383+0000 7fdac0a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdabc103290 0x7fdabc1a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:26.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.383+0000 7fdac0a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc10d340 con 0x7fdabc103290 2024-03-18T00:07:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.383+0000 7fdac0a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc10d1c0 con 0x7fdabc1016e0 2024-03-18T00:07:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.383+0000 7fdac0a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc10d4c0 con 0x7fdabc1028d0 2024-03-18T00:07:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.383+0000 7fdabaa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdabc1028d0 0x7fdabc19a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.383+0000 7fdabb400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdabc1016e0 0x7fdabc199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.383+0000 7fdabbe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdabc103290 0x7fdabc1a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.383+0000 7fdabaa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdabc1028d0 0x7fdabc19a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:39912/0 (socket says 172.21.15.12:39912) 2024-03-18T00:07:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.383+0000 7fdabaa00640 1 -- 172.21.15.12:0/1577003162 learned_addr learned my addr 172.21.15.12:0/1577003162 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.384+0000 7fdabb400640 1 -- 172.21.15.12:0/1577003162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdabc1028d0 msgr2=0x7fdabc19a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.384+0000 7fdabb400640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdabc1028d0 0x7fdabc19a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.384+0000 7fdabb400640 1 -- 172.21.15.12:0/1577003162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdabc103290 msgr2=0x7fdabc1a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.384+0000 7fdabb400640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdabc103290 0x7fdabc1a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.384+0000 7fdabb400640 1 -- 172.21.15.12:0/1577003162 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc1a1770 con 0x7fdabc1016e0 2024-03-18T00:07:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.384+0000 7fdabaa00640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdabc1028d0 0x7fdabc19a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:07:26.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.384+0000 7fdabbe00640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdabc103290 0x7fdabc1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:07:26.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.384+0000 7fdabb400640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdabc1016e0 0x7fdabc199b50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdaac00e9c0 tx=0x7fdaac00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:26.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.384+0000 7fdab3e00640 1 -- 172.21.15.12:0/1577003162 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac00cd80 con 0x7fdabc1016e0 2024-03-18T00:07:26.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.384+0000 7fdac0a00640 1 -- 172.21.15.12:0/1577003162 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdabc1a1a60 con 0x7fdabc1016e0 2024-03-18T00:07:26.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.384+0000 7fdab3e00640 1 -- 172.21.15.12:0/1577003162 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaac010910 con 0x7fdabc1016e0 2024-03-18T00:07:26.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.384+0000 7fdab3e00640 1 -- 172.21.15.12:0/1577003162 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac018980 con 0x7fdabc1016e0 2024-03-18T00:07:26.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.385+0000 7fdac0a00640 1 -- 172.21.15.12:0/1577003162 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdabc1a1fb0 con 0x7fdabc1016e0 2024-03-18T00:07:26.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.386+0000 7fdac0a00640 1 -- 172.21.15.12:0/1577003162 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdabc0684f0 con 0x7fdabc1016e0 2024-03-18T00:07:26.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.388+0000 7fdab3e00640 1 -- 172.21.15.12:0/1577003162 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fdaac01f020 con 0x7fdabc1016e0 2024-03-18T00:07:26.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.389+0000 7fdab3e00640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fdaa00a2800 0x7fdaa00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:26.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.389+0000 7fdab3e00640 1 -- 172.21.15.12:0/1577003162 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fdaac0cee30 con 0x7fdabc1016e0 2024-03-18T00:07:26.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.389+0000 7fdabaa00640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fdaa00a2800 0x7fdaa00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:26.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.390+0000 7fdabaa00640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fdaa00a2800 0x7fdaa00a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdabc19b060 tx=0x7fdaa8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:26.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.395+0000 7fdab3e00640 1 -- 172.21.15.12:0/1577003162 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdaac09baf0 con 0x7fdabc1016e0 2024-03-18T00:07:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.598+0000 7fdac0a00640 1 -- 172.21.15.12:0/1577003162 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdabc1a21e0 con 0x7fdabc1016e0 2024-03-18T00:07:26.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.599+0000 7fdab3e00640 1 -- 172.21.15.12:0/1577003162 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fdaac09b3f0 con 0x7fdabc1016e0 2024-03-18T00:07:26.599 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-18T00:07:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.602+0000 7fdac0a00640 1 -- 172.21.15.12:0/1577003162 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fdaa00a2800 msgr2=0x7fdaa00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.602+0000 7fdac0a00640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fdaa00a2800 0x7fdaa00a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdabc19b060 tx=0x7fdaa8005f90 comp rx=0 tx=0).stop 2024-03-18T00:07:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.602+0000 7fdac0a00640 1 -- 172.21.15.12:0/1577003162 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdabc1016e0 msgr2=0x7fdabc199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.602+0000 7fdac0a00640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdabc1016e0 0x7fdabc199b50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdaac00e9c0 tx=0x7fdaac00ee80 comp rx=0 tx=0).stop 2024-03-18T00:07:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.602+0000 7fdac0a00640 1 -- 172.21.15.12:0/1577003162 shutdown_connections 2024-03-18T00:07:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.602+0000 7fdac0a00640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fdaa00a2800 0x7fdaa00a4cc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.602+0000 7fdac0a00640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdabc103290 0x7fdabc1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.602+0000 7fdac0a00640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdabc1028d0 0x7fdabc19a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.602+0000 7fdac0a00640 1 --2- 172.21.15.12:0/1577003162 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdabc1016e0 0x7fdabc199b50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.602+0000 7fdac0a00640 1 -- 172.21.15.12:0/1577003162 >> 172.21.15.12:0/1577003162 conn(0x7fdabc0fcc50 msgr2=0x7fdabc073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.602+0000 7fdac0a00640 1 -- 172.21.15.12:0/1577003162 shutdown_connections 2024-03-18T00:07:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.602+0000 7fdac0a00640 1 -- 172.21.15.12:0/1577003162 wait complete. 2024-03-18T00:07:26.622 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T00:07:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.839+0000 7f52df400640 1 -- 172.21.15.12:0/559714778 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52d81055d0 msgr2=0x7f52d810be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.839+0000 7f52df400640 1 --2- 172.21.15.12:0/559714778 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52d81055d0 0x7f52d810be60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f52d0009ca0 tx=0x7f52d002c100 comp rx=0 tx=0).stop 2024-03-18T00:07:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.839+0000 7f52df400640 1 -- 172.21.15.12:0/559714778 shutdown_connections 2024-03-18T00:07:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.839+0000 7f52df400640 1 --2- 172.21.15.12:0/559714778 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52d81055d0 0x7f52d810be60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.839+0000 7f52df400640 1 --2- 172.21.15.12:0/559714778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d8104c10 0x7f52d8105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.839+0000 7f52df400640 1 --2- 172.21.15.12:0/559714778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52d8103a20 0x7f52d8103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.839+0000 7f52df400640 1 -- 172.21.15.12:0/559714778 >> 172.21.15.12:0/559714778 conn(0x7f52d80fef90 msgr2=0x7f52d81013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.839+0000 7f52df400640 1 -- 172.21.15.12:0/559714778 shutdown_connections 2024-03-18T00:07:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.839+0000 7f52df400640 1 -- 172.21.15.12:0/559714778 wait complete. 2024-03-18T00:07:26.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.840+0000 7f52df400640 1 Processor -- start 2024-03-18T00:07:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.840+0000 7f52df400640 1 -- start start 2024-03-18T00:07:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.840+0000 7f52df400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d8103a20 0x7f52d8199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52df400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52d8104c10 0x7f52d819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52df400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52d81055d0 0x7f52d81a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52df400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f52d810f680 con 0x7f52d8104c10 2024-03-18T00:07:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52df400640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f52d810f500 con 0x7f52d81055d0 2024-03-18T00:07:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52df400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f52d810f800 con 0x7f52d8103a20 2024-03-18T00:07:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52dca00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d8103a20 0x7f52d8199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52dca00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d8103a20 0x7f52d8199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:39932/0 (socket says 172.21.15.12:39932) 2024-03-18T00:07:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52dca00640 1 -- 172.21.15.12:0/3986487622 learned_addr learned my addr 172.21.15.12:0/3986487622 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52d7e00640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52d8104c10 0x7f52d819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:26.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52dd400640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52d81055d0 0x7f52d81a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:26.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52dca00640 1 -- 172.21.15.12:0/3986487622 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52d81055d0 msgr2=0x7f52d81a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:26.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52dca00640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52d81055d0 0x7f52d81a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52dca00640 1 -- 172.21.15.12:0/3986487622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52d8104c10 msgr2=0x7f52d819a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:26.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52dca00640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52d8104c10 0x7f52d819a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:26.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.841+0000 7f52dca00640 1 -- 172.21.15.12:0/3986487622 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d81a18d0 con 0x7f52d8103a20 2024-03-18T00:07:26.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.842+0000 7f52dca00640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d8103a20 0x7f52d8199c90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f52cc00e9c0 tx=0x7f52cc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:26.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.842+0000 7f52d5600640 1 -- 172.21.15.12:0/3986487622 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52cc00cd80 con 0x7f52d8103a20 2024-03-18T00:07:26.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.842+0000 7f52d5600640 1 -- 172.21.15.12:0/3986487622 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52cc0035f0 con 0x7f52d8103a20 2024-03-18T00:07:26.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.842+0000 7f52df400640 1 -- 172.21.15.12:0/3986487622 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52d81a1bc0 con 0x7f52d8103a20 2024-03-18T00:07:26.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.842+0000 7f52d5600640 1 -- 172.21.15.12:0/3986487622 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52cc010690 con 0x7f52d8103a20 2024-03-18T00:07:26.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.842+0000 7f52df400640 1 -- 172.21.15.12:0/3986487622 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52d81a2110 con 0x7f52d8103a20 2024-03-18T00:07:26.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.843+0000 7f52df400640 1 -- 172.21.15.12:0/3986487622 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52d810ace0 con 0x7f52d8103a20 2024-03-18T00:07:26.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.845+0000 7f52d5600640 1 -- 172.21.15.12:0/3986487622 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f52cc0037a0 con 0x7f52d8103a20 2024-03-18T00:07:26.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.846+0000 7f52d5600640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f52c00a2800 0x7f52c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:26.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.846+0000 7f52d5600640 1 -- 172.21.15.12:0/3986487622 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f52cc0cf1f0 con 0x7f52d8103a20 2024-03-18T00:07:26.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.847+0000 7f52d7e00640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f52c00a2800 0x7f52c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:26.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.847+0000 7f52d7e00640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f52c00a2800 0x7f52c00a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f52d819b190 tx=0x7f52c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:26.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:26.850+0000 7f52d5600640 1 -- 172.21.15.12:0/3986487622 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f52cc09beb0 con 0x7f52d8103a20 2024-03-18T00:07:26.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:26 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2346137895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:07:26.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:26 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1577003162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:07:27.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:26 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2346137895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:07:27.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:26 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1577003162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:07:27.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:26 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2346137895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:07:27.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:26 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1577003162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:07:27.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.154+0000 7f52df400640 1 -- 172.21.15.12:0/3986487622 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f52d8062480 con 0x7f52d8103a20 2024-03-18T00:07:27.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.155+0000 7f52d5600640 1 -- 172.21.15.12:0/3986487622 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f52cc09b7b0 con 0x7f52d8103a20 2024-03-18T00:07:27.155 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-03-18T00:07:27.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.157+0000 7f52df400640 1 -- 172.21.15.12:0/3986487622 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f52c00a2800 msgr2=0x7f52c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:27.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.157+0000 7f52df400640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f52c00a2800 0x7f52c00a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f52d819b190 tx=0x7f52c4005f90 comp rx=0 tx=0).stop 2024-03-18T00:07:27.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.157+0000 7f52df400640 1 -- 172.21.15.12:0/3986487622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d8103a20 msgr2=0x7f52d8199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:27.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.157+0000 7f52df400640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d8103a20 0x7f52d8199c90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f52cc00e9c0 tx=0x7f52cc00ee80 comp rx=0 tx=0).stop 2024-03-18T00:07:27.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.157+0000 7f52df400640 1 -- 172.21.15.12:0/3986487622 shutdown_connections 2024-03-18T00:07:27.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.157+0000 7f52df400640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f52c00a2800 0x7f52c00a4cc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.157+0000 7f52df400640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52d81055d0 0x7f52d81a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.157+0000 7f52df400640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52d8104c10 0x7f52d819a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.157+0000 7f52df400640 1 --2- 172.21.15.12:0/3986487622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52d8103a20 0x7f52d8199c90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.158+0000 7f52df400640 1 -- 172.21.15.12:0/3986487622 >> 172.21.15.12:0/3986487622 conn(0x7f52d80fef90 msgr2=0x7f52d8107eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:27.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.158+0000 7f52df400640 1 -- 172.21.15.12:0/3986487622 shutdown_connections 2024-03-18T00:07:27.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.158+0000 7f52df400640 1 -- 172.21.15.12:0/3986487622 wait complete. 2024-03-18T00:07:27.176 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T00:07:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.407+0000 7f3f4fc00640 1 -- 172.21.15.12:0/419896910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f481033a0 msgr2=0x7f3f48109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.407+0000 7f3f4fc00640 1 --2- 172.21.15.12:0/419896910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f481033a0 0x7f3f48109c30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3f40009ca0 tx=0x7f3f4002c0f0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.407+0000 7f3f4fc00640 1 -- 172.21.15.12:0/419896910 shutdown_connections 2024-03-18T00:07:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.407+0000 7f3f4fc00640 1 --2- 172.21.15.12:0/419896910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f481033a0 0x7f3f48109c30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.407+0000 7f3f4fc00640 1 --2- 172.21.15.12:0/419896910 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f481029e0 0x7f3f48102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.407+0000 7f3f4fc00640 1 --2- 172.21.15.12:0/419896910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f481017f0 0x7f3f48101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.407+0000 7f3f4fc00640 1 -- 172.21.15.12:0/419896910 >> 172.21.15.12:0/419896910 conn(0x7f3f480fcd80 msgr2=0x7f3f480ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.407+0000 7f3f4fc00640 1 -- 172.21.15.12:0/419896910 shutdown_connections 2024-03-18T00:07:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.407+0000 7f3f4fc00640 1 -- 172.21.15.12:0/419896910 wait complete. 2024-03-18T00:07:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.409+0000 7f3f4fc00640 1 Processor -- start 2024-03-18T00:07:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.409+0000 7f3f4fc00640 1 -- start start 2024-03-18T00:07:27.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.409+0000 7f3f4fc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f481017f0 0x7f3f48197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:27.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.409+0000 7f3f4fc00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f481029e0 0x7f3f48197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:27.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.409+0000 7f3f4fc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f481033a0 0x7f3f4819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:27.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.409+0000 7f3f4fc00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4810d450 con 0x7f3f481017f0 2024-03-18T00:07:27.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.409+0000 7f3f4fc00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4810d2d0 con 0x7f3f481029e0 2024-03-18T00:07:27.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.409+0000 7f3f4fc00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4810d5d0 con 0x7f3f481033a0 2024-03-18T00:07:27.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.410+0000 7f3f47e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f481029e0 0x7f3f48197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.410+0000 7f3f4d200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f481017f0 0x7f3f48197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.410+0000 7f3f47e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f481029e0 0x7f3f48197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60012/0 (socket says 172.21.15.12:60012) 2024-03-18T00:07:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.410+0000 7f3f47e00640 1 -- 172.21.15.12:0/983629600 learned_addr learned my addr 172.21.15.12:0/983629600 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.410+0000 7f3f4dc00640 1 --2- 172.21.15.12:0/983629600 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f481033a0 0x7f3f4819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.410+0000 7f3f47e00640 1 -- 172.21.15.12:0/983629600 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f481033a0 msgr2=0x7f3f4819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.410+0000 7f3f47e00640 1 --2- 172.21.15.12:0/983629600 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f481033a0 0x7f3f4819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.410+0000 7f3f47e00640 1 -- 172.21.15.12:0/983629600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f481017f0 msgr2=0x7f3f48197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.410+0000 7f3f47e00640 1 --2- 172.21.15.12:0/983629600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f481017f0 0x7f3f48197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.410+0000 7f3f47e00640 1 -- 172.21.15.12:0/983629600 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4819f6b0 con 0x7f3f481029e0 2024-03-18T00:07:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.410+0000 7f3f47e00640 1 --2- 172.21.15.12:0/983629600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f481029e0 0x7f3f48197fd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400d950 tx=0x7f3f3400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.411+0000 7f3f45600640 1 -- 172.21.15.12:0/983629600 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34017030 con 0x7f3f481029e0 2024-03-18T00:07:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.411+0000 7f3f45600640 1 -- 172.21.15.12:0/983629600 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f34004020 con 0x7f3f481029e0 2024-03-18T00:07:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.411+0000 7f3f4fc00640 1 -- 172.21.15.12:0/983629600 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4819f9a0 con 0x7f3f481029e0 2024-03-18T00:07:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.411+0000 7f3f45600640 1 -- 172.21.15.12:0/983629600 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f340047a0 con 0x7f3f481029e0 2024-03-18T00:07:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.411+0000 7f3f4fc00640 1 -- 172.21.15.12:0/983629600 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f4819ff40 con 0x7f3f481029e0 2024-03-18T00:07:27.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.412+0000 7f3f4fc00640 1 -- 172.21.15.12:0/983629600 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f48108b10 con 0x7f3f481029e0 2024-03-18T00:07:27.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.416+0000 7f3f45600640 1 -- 172.21.15.12:0/983629600 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f3f340041d0 con 0x7f3f481029e0 2024-03-18T00:07:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.417+0000 7f3f45600640 1 --2- 172.21.15.12:0/983629600 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3f300a2880 0x7f3f300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.417+0000 7f3f45600640 1 -- 172.21.15.12:0/983629600 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f3f340ce6b0 con 0x7f3f481029e0 2024-03-18T00:07:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.417+0000 7f3f4d200640 1 --2- 172.21.15.12:0/983629600 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3f300a2880 0x7f3f300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.418+0000 7f3f4d200640 1 --2- 172.21.15.12:0/983629600 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3f300a2880 0x7f3f300a4d40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3f48102840 tx=0x7f3f3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:27.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.425+0000 7f3f45600640 1 -- 172.21.15.12:0/983629600 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3f3409b3b0 con 0x7f3f481029e0 2024-03-18T00:07:27.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.630+0000 7f3f4fc00640 1 -- 172.21.15.12:0/983629600 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3f480623e0 con 0x7f3f481029e0 2024-03-18T00:07:27.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.631+0000 7f3f45600640 1 -- 172.21.15.12:0/983629600 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f3f3409acb0 con 0x7f3f481029e0 2024-03-18T00:07:27.632 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-03-18T00:07:27.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.633+0000 7f3f4fc00640 1 -- 172.21.15.12:0/983629600 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3f300a2880 msgr2=0x7f3f300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:27.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.633+0000 7f3f4fc00640 1 --2- 172.21.15.12:0/983629600 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3f300a2880 0x7f3f300a4d40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3f48102840 tx=0x7f3f3c005f90 comp rx=0 tx=0).stop 2024-03-18T00:07:27.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.633+0000 7f3f4fc00640 1 -- 172.21.15.12:0/983629600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f481029e0 msgr2=0x7f3f48197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:27.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.633+0000 7f3f4fc00640 1 --2- 172.21.15.12:0/983629600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f481029e0 0x7f3f48197fd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400d950 tx=0x7f3f3400de10 comp rx=0 tx=0).stop 2024-03-18T00:07:27.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.634+0000 7f3f4fc00640 1 -- 172.21.15.12:0/983629600 shutdown_connections 2024-03-18T00:07:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.634+0000 7f3f4fc00640 1 --2- 172.21.15.12:0/983629600 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f3f300a2880 0x7f3f300a4d40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.634+0000 7f3f4fc00640 1 --2- 172.21.15.12:0/983629600 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f481033a0 0x7f3f4819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.634+0000 7f3f4fc00640 1 --2- 172.21.15.12:0/983629600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f481029e0 0x7f3f48197fd0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.634+0000 7f3f4fc00640 1 --2- 172.21.15.12:0/983629600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f481017f0 0x7f3f48197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.634+0000 7f3f4fc00640 1 -- 172.21.15.12:0/983629600 >> 172.21.15.12:0/983629600 conn(0x7f3f480fcd80 msgr2=0x7f3f48105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.634+0000 7f3f4fc00640 1 -- 172.21.15.12:0/983629600 shutdown_connections 2024-03-18T00:07:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.634+0000 7f3f4fc00640 1 -- 172.21.15.12:0/983629600 wait complete. 2024-03-18T00:07:27.654 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T00:07:27.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.877+0000 7f148f400640 1 -- 172.21.15.12:0/3516111871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1488103a20 msgr2=0x7f1488103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:27.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.877+0000 7f148f400640 1 --2- 172.21.15.12:0/3516111871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1488103a20 0x7f1488103e40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f147c009900 tx=0x7f147c02bc40 comp rx=0 tx=0).stop 2024-03-18T00:07:27.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.877+0000 7f148f400640 1 -- 172.21.15.12:0/3516111871 shutdown_connections 2024-03-18T00:07:27.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.877+0000 7f148f400640 1 --2- 172.21.15.12:0/3516111871 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14881055d0 0x7f148810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.877+0000 7f148f400640 1 --2- 172.21.15.12:0/3516111871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1488104c10 0x7f1488105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.877+0000 7f148f400640 1 --2- 172.21.15.12:0/3516111871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1488103a20 0x7f1488103e40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.877+0000 7f148f400640 1 -- 172.21.15.12:0/3516111871 >> 172.21.15.12:0/3516111871 conn(0x7f14880fef90 msgr2=0x7f14881013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:27.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.877+0000 7f148f400640 1 -- 172.21.15.12:0/3516111871 shutdown_connections 2024-03-18T00:07:27.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.877+0000 7f148f400640 1 -- 172.21.15.12:0/3516111871 wait complete. 2024-03-18T00:07:27.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.878+0000 7f148f400640 1 Processor -- start 2024-03-18T00:07:27.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f148f400640 1 -- start start 2024-03-18T00:07:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f148f400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1488103a20 0x7f1488199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f148f400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1488104c10 0x7f148819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f148f400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14881055d0 0x7f14881a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f148f400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f148810f670 con 0x7f1488103a20 2024-03-18T00:07:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f148f400640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f148810f4f0 con 0x7f14881055d0 2024-03-18T00:07:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f148f400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f148810f7f0 con 0x7f1488104c10 2024-03-18T00:07:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f148ca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1488103a20 0x7f1488199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f1487e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1488104c10 0x7f148819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f148d400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14881055d0 0x7f14881a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f148ca00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1488103a20 0x7f1488199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47006/0 (socket says 172.21.15.12:47006) 2024-03-18T00:07:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f148ca00640 1 -- 172.21.15.12:0/4019013196 learned_addr learned my addr 172.21.15.12:0/4019013196 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f1487e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1488104c10 0x7f148819a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:39974/0 (socket says 172.21.15.12:39974) 2024-03-18T00:07:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.879+0000 7f148d400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14881055d0 0x7f14881a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60020/0 (socket says 172.21.15.12:60020) 2024-03-18T00:07:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.880+0000 7f148d400640 1 -- 172.21.15.12:0/4019013196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1488104c10 msgr2=0x7f148819a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.880+0000 7f148d400640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1488104c10 0x7f148819a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.880+0000 7f148d400640 1 -- 172.21.15.12:0/4019013196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1488103a20 msgr2=0x7f1488199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.880+0000 7f148d400640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1488103a20 0x7f1488199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.880+0000 7f148d400640 1 -- 172.21.15.12:0/4019013196 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14881a19c0 con 0x7f14881055d0 2024-03-18T00:07:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.880+0000 7f1487e00640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1488104c10 0x7f148819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:07:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.880+0000 7f148ca00640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1488103a20 0x7f1488199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:07:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.880+0000 7f148d400640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14881055d0 0x7f14881a1320 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f148000da10 tx=0x7f148000ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:27.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.880+0000 7f1485600640 1 -- 172.21.15.12:0/4019013196 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148000bd70 con 0x7f14881055d0 2024-03-18T00:07:27.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.880+0000 7f1485600640 1 -- 172.21.15.12:0/4019013196 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1480003e50 con 0x7f14881055d0 2024-03-18T00:07:27.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.881+0000 7f1485600640 1 -- 172.21.15.12:0/4019013196 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14800106a0 con 0x7f14881055d0 2024-03-18T00:07:27.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.881+0000 7f148f400640 1 -- 172.21.15.12:0/4019013196 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14881a1cb0 con 0x7f14881055d0 2024-03-18T00:07:27.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.881+0000 7f148f400640 1 -- 172.21.15.12:0/4019013196 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14881a21b0 con 0x7f14881055d0 2024-03-18T00:07:27.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.882+0000 7f148f400640 1 -- 172.21.15.12:0/4019013196 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f148810ade0 con 0x7f14881055d0 2024-03-18T00:07:27.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.886+0000 7f1485600640 1 -- 172.21.15.12:0/4019013196 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f1480003970 con 0x7f14881055d0 2024-03-18T00:07:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.887+0000 7f1485600640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14700a2880 0x7f14700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.887+0000 7f1485600640 1 -- 172.21.15.12:0/4019013196 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f14800cf230 con 0x7f14881055d0 2024-03-18T00:07:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.887+0000 7f148ca00640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14700a2880 0x7f14700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.888+0000 7f148ca00640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14700a2880 0x7f14700a4d40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f147c006010 tx=0x7f147c005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:27.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:27.892+0000 7f1485600640 1 -- 172.21.15.12:0/4019013196 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f148009bef0 con 0x7f14881055d0 2024-03-18T00:07:27.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:27 smithi012 ceph-mon[31541]: pgmap v700: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-18T00:07:27.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:27 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3986487622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:07:27.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:27 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/983629600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T00:07:28.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:27 smithi079 ceph-mon[36241]: pgmap v700: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-18T00:07:28.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:27 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3986487622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:07:28.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:27 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/983629600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T00:07:28.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:27 smithi191 ceph-mon[36708]: pgmap v700: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-18T00:07:28.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:27 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3986487622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:07:28.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:27 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/983629600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T00:07:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.115+0000 7f148f400640 1 -- 172.21.15.12:0/4019013196 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f148802d1b0 con 0x7f14881055d0 2024-03-18T00:07:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.115+0000 7f1485600640 1 -- 172.21.15.12:0/4019013196 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v94) ==== 81+0+590 (secure 0 0 0) 0x7f148009b7f0 con 0x7f14881055d0 2024-03-18T00:07:28.116 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/2526360059 2024-03-18T23:46:53.955890+0000 2024-03-18T00:07:28.116 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/2336882461 2024-03-18T23:46:53.955890+0000 2024-03-18T00:07:28.116 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/1563339918 2024-03-18T23:46:19.706199+0000 2024-03-18T00:07:28.116 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1976882257 2024-03-18T23:46:53.955890+0000 2024-03-18T00:07:28.116 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/501542164 2024-03-18T23:46:53.955890+0000 2024-03-18T00:07:28.116 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/654354506 2024-03-18T23:46:19.706199+0000 2024-03-18T00:07:28.116 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/1563339918 2024-03-18T23:46:19.706199+0000 2024-03-18T00:07:28.116 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1974593432 2024-03-18T23:46:19.706199+0000 2024-03-18T00:07:28.116 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/2526360059 2024-03-18T23:46:53.955890+0000 2024-03-18T00:07:28.116 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1690054132 2024-03-18T23:46:19.706199+0000 2024-03-18T00:07:28.117 INFO:teuthology.orchestra.run.smithi012.stderr:listed 10 entries 2024-03-18T00:07:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.118+0000 7f148f400640 1 -- 172.21.15.12:0/4019013196 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14700a2880 msgr2=0x7f14700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.118+0000 7f148f400640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14700a2880 0x7f14700a4d40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f147c006010 tx=0x7f147c005b50 comp rx=0 tx=0).stop 2024-03-18T00:07:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.118+0000 7f148f400640 1 -- 172.21.15.12:0/4019013196 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14881055d0 msgr2=0x7f14881a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.118+0000 7f148f400640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14881055d0 0x7f14881a1320 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f148000da10 tx=0x7f148000ded0 comp rx=0 tx=0).stop 2024-03-18T00:07:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.118+0000 7f148f400640 1 -- 172.21.15.12:0/4019013196 shutdown_connections 2024-03-18T00:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.118+0000 7f148f400640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f14700a2880 0x7f14700a4d40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.118+0000 7f148f400640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14881055d0 0x7f14881a1320 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.118+0000 7f148f400640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1488104c10 0x7f148819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.118+0000 7f148f400640 1 --2- 172.21.15.12:0/4019013196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1488103a20 0x7f1488199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.118+0000 7f148f400640 1 -- 172.21.15.12:0/4019013196 >> 172.21.15.12:0/4019013196 conn(0x7f14880fef90 msgr2=0x7f1488107fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.119+0000 7f148f400640 1 -- 172.21.15.12:0/4019013196 shutdown_connections 2024-03-18T00:07:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.119+0000 7f148f400640 1 -- 172.21.15.12:0/4019013196 wait complete. 2024-03-18T00:07:28.139 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-18T00:07:28.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:07:28.140 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-18T00:07:28.372 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-03-18T00:07:28.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:07:28.372 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T00:07:28.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:07:28.454 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-03-18T00:07:28.532 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T00:07:28.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:07:28.533 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T00:07:28.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:07:28.668 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-03-18T00:07:28.796 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-18T00:07:28.809 INFO:tasks.cephadm:Teardown begin 2024-03-18T00:07:28.809 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T00:07:28.810 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:07:28.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:28 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/4019013196' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T00:07:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.990+0000 7fc2b8a00640 1 -- 172.21.15.12:0/82135499 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2b4104bf0 msgr2=0x7fc2b4105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.990+0000 7fc2b8a00640 1 --2- 172.21.15.12:0/82135499 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2b4104bf0 0x7fc2b4105070 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009900 tx=0x7fc29c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:07:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.990+0000 7fc2b8a00640 1 -- 172.21.15.12:0/82135499 shutdown_connections 2024-03-18T00:07:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.990+0000 7fc2b8a00640 1 --2- 172.21.15.12:0/82135499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b41055b0 0x7fc2b410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.990+0000 7fc2b8a00640 1 --2- 172.21.15.12:0/82135499 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2b4104bf0 0x7fc2b4105070 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.990+0000 7fc2b8a00640 1 --2- 172.21.15.12:0/82135499 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4103a00 0x7fc2b4103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.990+0000 7fc2b8a00640 1 -- 172.21.15.12:0/82135499 >> 172.21.15.12:0/82135499 conn(0x7fc2b40fef90 msgr2=0x7fc2b41013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.990+0000 7fc2b8a00640 1 -- 172.21.15.12:0/82135499 shutdown_connections 2024-03-18T00:07:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.990+0000 7fc2b8a00640 1 -- 172.21.15.12:0/82135499 wait complete. 2024-03-18T00:07:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.991+0000 7fc2b8a00640 1 Processor -- start 2024-03-18T00:07:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.992+0000 7fc2b8a00640 1 -- start start 2024-03-18T00:07:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.992+0000 7fc2b8a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4103a00 0x7fc2b4199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.992+0000 7fc2b8a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2b4104bf0 0x7fc2b419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.992+0000 7fc2b8a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b41055b0 0x7fc2b41a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.992+0000 7fc2b8a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b410f6b0 con 0x7fc2b4103a00 2024-03-18T00:07:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.992+0000 7fc2b8a00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b410f530 con 0x7fc2b4104bf0 2024-03-18T00:07:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.992+0000 7fc2b8a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b410f830 con 0x7fc2b41055b0 2024-03-18T00:07:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.992+0000 7fc2a9400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2b4104bf0 0x7fc2b419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.992+0000 7fc2b1e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4103a00 0x7fc2b4199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.992+0000 7fc2a9400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2b4104bf0 0x7fc2b419a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60038/0 (socket says 172.21.15.12:60038) 2024-03-18T00:07:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.992+0000 7fc2a9400640 1 -- 172.21.15.12:0/1399972290 learned_addr learned my addr 172.21.15.12:0/1399972290 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.992+0000 7fc2b2800640 1 --2- 172.21.15.12:0/1399972290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b41055b0 0x7fc2b41a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.993+0000 7fc2a9400640 1 -- 172.21.15.12:0/1399972290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b41055b0 msgr2=0x7fc2b41a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.993+0000 7fc2a9400640 1 --2- 172.21.15.12:0/1399972290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b41055b0 0x7fc2b41a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.993+0000 7fc2a9400640 1 -- 172.21.15.12:0/1399972290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4103a00 msgr2=0x7fc2b4199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.993+0000 7fc2a9400640 1 --2- 172.21.15.12:0/1399972290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4103a00 0x7fc2b4199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.993+0000 7fc2a9400640 1 -- 172.21.15.12:0/1399972290 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b41a1900 con 0x7fc2b4104bf0 2024-03-18T00:07:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.993+0000 7fc2a9400640 1 --2- 172.21.15.12:0/1399972290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2b4104bf0 0x7fc2b419a200 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc29c005ab0 tx=0x7fc29c02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.993+0000 7fc2ab400640 1 -- 172.21.15.12:0/1399972290 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29c043020 con 0x7fc2b4104bf0 2024-03-18T00:07:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.993+0000 7fc2ab400640 1 -- 172.21.15.12:0/1399972290 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc29c0054e0 con 0x7fc2b4104bf0 2024-03-18T00:07:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.993+0000 7fc2b8a00640 1 -- 172.21.15.12:0/1399972290 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b41a1b90 con 0x7fc2b4104bf0 2024-03-18T00:07:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.993+0000 7fc2ab400640 1 -- 172.21.15.12:0/1399972290 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29c003ce0 con 0x7fc2b4104bf0 2024-03-18T00:07:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.994+0000 7fc2b8a00640 1 -- 172.21.15.12:0/1399972290 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b41a20e0 con 0x7fc2b4104bf0 2024-03-18T00:07:28.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.994+0000 7fc2b8a00640 1 -- 172.21.15.12:0/1399972290 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b410ad20 con 0x7fc2b4104bf0 2024-03-18T00:07:28.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:28.998+0000 7fc2ab400640 1 -- 172.21.15.12:0/1399972290 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fc29c046020 con 0x7fc2b4104bf0 2024-03-18T00:07:29.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.000+0000 7fc2ab400640 1 --2- 172.21.15.12:0/1399972290 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc2980a2800 0x7fc2980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:29.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.000+0000 7fc2ab400640 1 -- 172.21.15.12:0/1399972290 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fc29c0f12b0 con 0x7fc2b4104bf0 2024-03-18T00:07:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.000+0000 7fc2b1e00640 1 --2- 172.21.15.12:0/1399972290 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc2980a2800 0x7fc2980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.001+0000 7fc2b1e00640 1 --2- 172.21.15.12:0/1399972290 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc2980a2800 0x7fc2980a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc2b419ab90 tx=0x7fc2a400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:29.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.005+0000 7fc2ab400640 1 -- 172.21.15.12:0/1399972290 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc29c04d020 con 0x7fc2b4104bf0 2024-03-18T00:07:29.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:28 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/4019013196' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T00:07:29.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:28 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/4019013196' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T00:07:29.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.241+0000 7fc2b8a00640 1 -- 172.21.15.12:0/1399972290 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2b402d1b0 con 0x7fc2b4104bf0 2024-03-18T00:07:29.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.243+0000 7fc2ab400640 1 -- 172.21.15.12:0/1399972290 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11951 (secure 0 0 0) 0x7fc29c0bdf70 con 0x7fc2b4104bf0 2024-03-18T00:07:29.243 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-18T00:07:29.243 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":49,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-17T23:54:56.312554+0000","modified":"2024-03-18T00:07:24.085304+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24575,"mds_2":24530,"mds_3":24529,"mds_4":24514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.12:6841/2073607248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":2073607248},{"type":"v1","addr":"172.21.15.12:6841","nonce":2073607248}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/2478742711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":2478742711},{"type":"v1","addr":"172.21.15.79:6839","nonce":2478742711}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24514,24529,24530,24575,24587]},"id":1}]} 2024-03-18T00:07:29.244 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 49 2024-03-18T00:07:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.245+0000 7fc2b8a00640 1 -- 172.21.15.12:0/1399972290 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc2980a2800 msgr2=0x7fc2980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.245+0000 7fc2b8a00640 1 --2- 172.21.15.12:0/1399972290 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc2980a2800 0x7fc2980a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc2b419ab90 tx=0x7fc2a400b410 comp rx=0 tx=0).stop 2024-03-18T00:07:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.245+0000 7fc2b8a00640 1 -- 172.21.15.12:0/1399972290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2b4104bf0 msgr2=0x7fc2b419a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.245+0000 7fc2b8a00640 1 --2- 172.21.15.12:0/1399972290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2b4104bf0 0x7fc2b419a200 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc29c005ab0 tx=0x7fc29c02c6c0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.245+0000 7fc2b8a00640 1 -- 172.21.15.12:0/1399972290 shutdown_connections 2024-03-18T00:07:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.245+0000 7fc2b8a00640 1 --2- 172.21.15.12:0/1399972290 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fc2980a2800 0x7fc2980a4cc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.245+0000 7fc2b8a00640 1 --2- 172.21.15.12:0/1399972290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b41055b0 0x7fc2b41a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.245+0000 7fc2b8a00640 1 --2- 172.21.15.12:0/1399972290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2b4104bf0 0x7fc2b419a200 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.245+0000 7fc2b8a00640 1 --2- 172.21.15.12:0/1399972290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4103a00 0x7fc2b4199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.245+0000 7fc2b8a00640 1 -- 172.21.15.12:0/1399972290 >> 172.21.15.12:0/1399972290 conn(0x7fc2b40fef90 msgr2=0x7fc2b4107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:29.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.246+0000 7fc2b8a00640 1 -- 172.21.15.12:0/1399972290 shutdown_connections 2024-03-18T00:07:29.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.246+0000 7fc2b8a00640 1 -- 172.21.15.12:0/1399972290 wait complete. 2024-03-18T00:07:29.264 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.459+0000 7f7166000640 1 -- 172.21.15.12:0/707401995 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f71601031e0 msgr2=0x7f7160109a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.459+0000 7f7166000640 1 --2- 172.21.15.12:0/707401995 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f71601031e0 0x7f7160109a70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7158009900 tx=0x7f715802bc30 comp rx=0 tx=0).stop 2024-03-18T00:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.459+0000 7f7166000640 1 -- 172.21.15.12:0/707401995 shutdown_connections 2024-03-18T00:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.459+0000 7f7166000640 1 --2- 172.21.15.12:0/707401995 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f71601031e0 0x7f7160109a70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.459+0000 7f7166000640 1 --2- 172.21.15.12:0/707401995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160102820 0x7f7160102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.459+0000 7f7166000640 1 --2- 172.21.15.12:0/707401995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7160101630 0x7f7160101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.459+0000 7f7166000640 1 -- 172.21.15.12:0/707401995 >> 172.21.15.12:0/707401995 conn(0x7f71600fcbc0 msgr2=0x7f71600fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.459+0000 7f7166000640 1 -- 172.21.15.12:0/707401995 shutdown_connections 2024-03-18T00:07:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.459+0000 7f7166000640 1 -- 172.21.15.12:0/707401995 wait complete. 2024-03-18T00:07:29.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.460+0000 7f7166000640 1 Processor -- start 2024-03-18T00:07:29.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.460+0000 7f7166000640 1 -- start start 2024-03-18T00:07:29.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.460+0000 7f7166000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7160101630 0x7f7160075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.460+0000 7f7166000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7160102820 0x7f7160073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.460+0000 7f7166000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71601031e0 0x7f7160074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.460+0000 7f7166000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f716010d300 con 0x7f71601031e0 2024-03-18T00:07:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.460+0000 7f7166000640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f716010d180 con 0x7f7160102820 2024-03-18T00:07:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.460+0000 7f7166000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f716010d480 con 0x7f7160101630 2024-03-18T00:07:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.461+0000 7f7164c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7160101630 0x7f7160075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.461+0000 7f715fe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7160102820 0x7f7160073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.461+0000 7f7164c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7160101630 0x7f7160075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:40016/0 (socket says 172.21.15.12:40016) 2024-03-18T00:07:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.461+0000 7f7164c00640 1 -- 172.21.15.12:0/1760404770 learned_addr learned my addr 172.21.15.12:0/1760404770 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.461+0000 7f7165600640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71601031e0 0x7f7160074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.461+0000 7f7164c00640 1 -- 172.21.15.12:0/1760404770 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7160102820 msgr2=0x7f7160073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.461+0000 7f7164c00640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7160102820 0x7f7160073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.461+0000 7f7164c00640 1 -- 172.21.15.12:0/1760404770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71601031e0 msgr2=0x7f7160074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.461+0000 7f7164c00640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71601031e0 0x7f7160074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.461+0000 7f7164c00640 1 -- 172.21.15.12:0/1760404770 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71600749a0 con 0x7f7160101630 2024-03-18T00:07:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.461+0000 7f7165600640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71601031e0 0x7f7160074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:07:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.461+0000 7f715fe00640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7160102820 0x7f7160073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:07:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.461+0000 7f7164c00640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7160101630 0x7f7160075590 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f715400eb70 tx=0x7f715400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.462+0000 7f715d600640 1 -- 172.21.15.12:0/1760404770 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7154003b70 con 0x7f7160101630 2024-03-18T00:07:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.462+0000 7f715d600640 1 -- 172.21.15.12:0/1760404770 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f715400ee70 con 0x7f7160101630 2024-03-18T00:07:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.462+0000 7f715d600640 1 -- 172.21.15.12:0/1760404770 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7154010830 con 0x7f7160101630 2024-03-18T00:07:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.462+0000 7f7166000640 1 -- 172.21.15.12:0/1760404770 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71601a19b0 con 0x7f7160101630 2024-03-18T00:07:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.462+0000 7f7166000640 1 -- 172.21.15.12:0/1760404770 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71601a1eb0 con 0x7f7160101630 2024-03-18T00:07:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.463+0000 7f7166000640 1 -- 172.21.15.12:0/1760404770 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7160108880 con 0x7f7160101630 2024-03-18T00:07:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.465+0000 7f715d600640 1 -- 172.21.15.12:0/1760404770 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f7154004a40 con 0x7f7160101630 2024-03-18T00:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.466+0000 7f715d600640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f71480a2880 0x7f71480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.466+0000 7f715d600640 1 -- 172.21.15.12:0/1760404770 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f71540cf660 con 0x7f7160101630 2024-03-18T00:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.466+0000 7f715fe00640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f71480a2880 0x7f71480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.467+0000 7f715fe00640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f71480a2880 0x7f71480a4d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7160075300 tx=0x7f714c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.468+0000 7f715d600640 1 -- 172.21.15.12:0/1760404770 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f715409c320 con 0x7f7160101630 2024-03-18T00:07:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.717+0000 7f7166000640 1 -- 172.21.15.12:0/1760404770 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f716002d370 con 0x7f7160101630 2024-03-18T00:07:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.718+0000 7f715d600640 1 -- 172.21.15.12:0/1760404770 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11951 (secure 0 0 0) 0x7f715409bc20 con 0x7f7160101630 2024-03-18T00:07:29.719 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-18T00:07:29.719 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14376,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/1091924746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":1091924746},{"type":"v1","addr":"172.21.15.12:6835","nonce":1091924746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14442,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/1516405819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1516405819},{"type":"v1","addr":"172.21.15.12:6839","nonce":1516405819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":49,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-17T23:54:56.312554+0000","modified":"2024-03-18T00:07:24.085304+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24575,"mds_2":24530,"mds_3":24529,"mds_4":24514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/1787103083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1787103083},{"type":"v1","addr":"172.21.15.191:6833","nonce":1787103083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6833/843208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":843208408},{"type":"v1","addr":"172.21.15.79:6833","nonce":843208408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.12:6837/1104379442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":1104379442},{"type":"v1","addr":"172.21.15.12:6837","nonce":1104379442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3192827956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3192827956},{"type":"v1","addr":"172.21.15.191:6835","nonce":3192827956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6835/4099928196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":4099928196},{"type":"v1","addr":"172.21.15.79:6835","nonce":4099928196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.191:6837/2977770216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2977770216},{"type":"v1","addr":"172.21.15.191:6837","nonce":2977770216}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6839/4148729123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4148729123},{"type":"v1","addr":"172.21.15.191:6839","nonce":4148729123}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.79:6837/3814468390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3814468390},{"type":"v1","addr":"172.21.15.79:6837","nonce":3814468390}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.12:6841/2073607248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":2073607248},{"type":"v1","addr":"172.21.15.12:6841","nonce":2073607248}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/2478742711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":2478742711},{"type":"v1","addr":"172.21.15.79:6839","nonce":2478742711}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24514,24529,24530,24575,24587]},"id":1}]} 2024-03-18T00:07:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 49 2024-03-18T00:07:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.721+0000 7f7166000640 1 -- 172.21.15.12:0/1760404770 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f71480a2880 msgr2=0x7f71480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.721+0000 7f7166000640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f71480a2880 0x7f71480a4d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7160075300 tx=0x7f714c00b410 comp rx=0 tx=0).stop 2024-03-18T00:07:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.721+0000 7f7166000640 1 -- 172.21.15.12:0/1760404770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7160101630 msgr2=0x7f7160075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.721+0000 7f7166000640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7160101630 0x7f7160075590 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f715400eb70 tx=0x7f715400c6a0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.721+0000 7f7166000640 1 -- 172.21.15.12:0/1760404770 shutdown_connections 2024-03-18T00:07:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.721+0000 7f7166000640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f71480a2880 0x7f71480a4d40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.721+0000 7f7166000640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71601031e0 0x7f7160074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.721+0000 7f7166000640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7160102820 0x7f7160073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.721+0000 7f7166000640 1 --2- 172.21.15.12:0/1760404770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7160101630 0x7f7160075590 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.721+0000 7f7166000640 1 -- 172.21.15.12:0/1760404770 >> 172.21.15.12:0/1760404770 conn(0x7f71600fcbc0 msgr2=0x7f7160105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.721+0000 7f7166000640 1 -- 172.21.15.12:0/1760404770 shutdown_connections 2024-03-18T00:07:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.721+0000 7f7166000640 1 -- 172.21.15.12:0/1760404770 wait complete. 2024-03-18T00:07:29.741 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:07:29.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.959+0000 7f1c7e000640 1 -- 172.21.15.12:0/1179949408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c78104a20 msgr2=0x7f1c78106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:29.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.959+0000 7f1c7e000640 1 --2- 172.21.15.12:0/1179949408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c78104a20 0x7f1c78106e10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1c64009900 tx=0x7f1c6402bc30 comp rx=0 tx=0).stop 2024-03-18T00:07:29.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.959+0000 7f1c7e000640 1 -- 172.21.15.12:0/1179949408 shutdown_connections 2024-03-18T00:07:29.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.959+0000 7f1c7e000640 1 --2- 172.21.15.12:0/1179949408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c78107350 0x7f1c78109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.959+0000 7f1c7e000640 1 --2- 172.21.15.12:0/1179949408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c78104a20 0x7f1c78106e10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.959+0000 7f1c7e000640 1 --2- 172.21.15.12:0/1179949408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c781020f0 0x7f1c781044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.959+0000 7f1c7e000640 1 -- 172.21.15.12:0/1179949408 >> 172.21.15.12:0/1179949408 conn(0x7f1c780fba60 msgr2=0x7f1c780fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:29.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.959+0000 7f1c7e000640 1 -- 172.21.15.12:0/1179949408 shutdown_connections 2024-03-18T00:07:29.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.959+0000 7f1c7e000640 1 -- 172.21.15.12:0/1179949408 wait complete. 2024-03-18T00:07:29.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.960+0000 7f1c7e000640 1 Processor -- start 2024-03-18T00:07:29.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.961+0000 7f1c7e000640 1 -- start start 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.961+0000 7f1c7e000640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c781020f0 0x7f1c78101a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.961+0000 7f1c7e000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c78104a20 0x7f1c781000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.961+0000 7f1c7e000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c78107350 0x7f1c781005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.961+0000 7f1c7e000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7810d2c0 con 0x7f1c78104a20 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.961+0000 7f1c7e000640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7810d140 con 0x7f1c781020f0 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.961+0000 7f1c7e000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7810d440 con 0x7f1c78107350 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.961+0000 7f1c7cc00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c781020f0 0x7f1c78101a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.961+0000 7f1c7d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c78107350 0x7f1c781005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.961+0000 7f1c77e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c78104a20 0x7f1c781000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.961+0000 7f1c7d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c78107350 0x7f1c781005e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:40038/0 (socket says 172.21.15.12:40038) 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.961+0000 7f1c7d600640 1 -- 172.21.15.12:0/58420602 learned_addr learned my addr 172.21.15.12:0/58420602 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.961+0000 7f1c7cc00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c781020f0 0x7f1c78101a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60090/0 (socket says 172.21.15.12:60090) 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.962+0000 7f1c7d600640 1 -- 172.21.15.12:0/58420602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c781020f0 msgr2=0x7f1c78101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:29.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.962+0000 7f1c7d600640 1 --2- 172.21.15.12:0/58420602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c781020f0 0x7f1c78101a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.962+0000 7f1c7d600640 1 -- 172.21.15.12:0/58420602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c78104a20 msgr2=0x7f1c781000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:29.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.962+0000 7f1c7d600640 1 --2- 172.21.15.12:0/58420602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c78104a20 0x7f1c781000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:29.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.962+0000 7f1c7d600640 1 -- 172.21.15.12:0/58420602 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c78100e70 con 0x7f1c78107350 2024-03-18T00:07:29.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.962+0000 7f1c7cc00640 1 --2- 172.21.15.12:0/58420602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c781020f0 0x7f1c78101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:07:29.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.962+0000 7f1c7d600640 1 --2- 172.21.15.12:0/58420602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c78107350 0x7f1c781005e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000eb10 tx=0x7f1c7000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:29.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.962+0000 7f1c75600640 1 -- 172.21.15.12:0/58420602 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c70004b20 con 0x7f1c78107350 2024-03-18T00:07:29.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.963+0000 7f1c7e000640 1 -- 172.21.15.12:0/58420602 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c781a19f0 con 0x7f1c78107350 2024-03-18T00:07:29.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.963+0000 7f1c75600640 1 -- 172.21.15.12:0/58420602 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c70004cc0 con 0x7f1c78107350 2024-03-18T00:07:29.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.963+0000 7f1c75600640 1 -- 172.21.15.12:0/58420602 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c70010760 con 0x7f1c78107350 2024-03-18T00:07:29.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.963+0000 7f1c7e000640 1 -- 172.21.15.12:0/58420602 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c781a1f10 con 0x7f1c78107350 2024-03-18T00:07:29.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.964+0000 7f1c7e000640 1 -- 172.21.15.12:0/58420602 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c780684f0 con 0x7f1c78107350 2024-03-18T00:07:29.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.966+0000 7f1c75600640 1 -- 172.21.15.12:0/58420602 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f1c70005020 con 0x7f1c78107350 2024-03-18T00:07:29.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.967+0000 7f1c75600640 1 --2- 172.21.15.12:0/58420602 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1c600a2800 0x7f1c600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:29.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.967+0000 7f1c75600640 1 -- 172.21.15.12:0/58420602 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f1c700cf560 con 0x7f1c78107350 2024-03-18T00:07:29.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.967+0000 7f1c7cc00640 1 --2- 172.21.15.12:0/58420602 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1c600a2800 0x7f1c600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:29.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.968+0000 7f1c7cc00640 1 --2- 172.21.15.12:0/58420602 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1c600a2800 0x7f1c600a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c00bfd0 tx=0x7f1c6c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:29.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:29.971+0000 7f1c75600640 1 -- 172.21.15.12:0/58420602 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1c7009c220 con 0x7f1c78107350 2024-03-18T00:07:30.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:29 smithi079 ceph-mon[36241]: pgmap v701: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T00:07:30.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:29 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1399972290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:07:30.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:29 smithi191 ceph-mon[36708]: pgmap v701: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T00:07:30.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:29 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1399972290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:07:30.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:29 smithi012 ceph-mon[31541]: pgmap v701: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T00:07:30.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:29 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1399972290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:07:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.206+0000 7f1c7e000640 1 -- 172.21.15.12:0/58420602 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c780623e0 con 0x7f1c78107350 2024-03-18T00:07:30.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.208+0000 7f1c75600640 1 -- 172.21.15.12:0/58420602 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24618 (secure 0 0 0) 0x7f1c7009bb20 con 0x7f1c78107350 2024-03-18T00:07:30.209 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-18T00:07:30.209 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":94,"fsid":"4558f3ea-e4b8-11ee-95c9-87774f69a715","created":"2024-03-17T23:45:45.898190+0000","modified":"2024-03-17T23:56:55.719277+0000","last_up_change":"2024-03-17T23:53:57.248586+0000","last_in_change":"2024-03-17T23:53:37.180979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:49:38.640282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:54:54.540345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:54:55.803926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:55:00.355066+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b91e248b-f111-4dc6-9dc8-07fd0195699a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6803","nonce":3127075279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6805","nonce":3127075279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6809","nonce":3127075279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3127075279},{"type":"v1","addr":"172.21.15.12:6807","nonce":3127075279}]},"public_addr":"172.21.15.12:6803/3127075279","cluster_addr":"172.21.15.12:6805/3127075279","heartbeat_back_addr":"172.21.15.12:6809/3127075279","heartbeat_front_addr":"172.21.15.12:6807/3127075279","state":["exists","up"]},{"osd":1,"uuid":"13bf3ff0-c8b3-4c24-9960-c6568256772f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6801","nonce":890307857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6803","nonce":890307857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6807","nonce":890307857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":890307857},{"type":"v1","addr":"172.21.15.79:6805","nonce":890307857}]},"public_addr":"172.21.15.79:6801/890307857","cluster_addr":"172.21.15.79:6803/890307857","heartbeat_back_addr":"172.21.15.79:6807/890307857","heartbeat_front_addr":"172.21.15.79:6805/890307857","state":["exists","up"]},{"osd":2,"uuid":"24aaf834-2a6b-4bb0-986b-3fe9525ecacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6801","nonce":936135993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6803","nonce":936135993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6807","nonce":936135993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":936135993},{"type":"v1","addr":"172.21.15.191:6805","nonce":936135993}]},"public_addr":"172.21.15.191:6801/936135993","cluster_addr":"172.21.15.191:6803/936135993","heartbeat_back_addr":"172.21.15.191:6807/936135993","heartbeat_front_addr":"172.21.15.191:6805/936135993","state":["exists","up"]},{"osd":3,"uuid":"ba4c0892-f619-45d4-a91a-00a2b0cc3d78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6811","nonce":769150575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6813","nonce":769150575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6817","nonce":769150575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":769150575},{"type":"v1","addr":"172.21.15.12:6815","nonce":769150575}]},"public_addr":"172.21.15.12:6811/769150575","cluster_addr":"172.21.15.12:6813/769150575","heartbeat_back_addr":"172.21.15.12:6817/769150575","heartbeat_front_addr":"172.21.15.12:6815/769150575","state":["exists","up"]},{"osd":4,"uuid":"ba1ba8fd-e3b7-4688-af8f-6cb7bec9bc4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6809","nonce":3381761360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6811","nonce":3381761360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6815","nonce":3381761360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3381761360},{"type":"v1","addr":"172.21.15.79:6813","nonce":3381761360}]},"public_addr":"172.21.15.79:6809/3381761360","cluster_addr":"172.21.15.79:6811/3381761360","heartbeat_back_addr":"172.21.15.79:6815/3381761360","heartbeat_front_addr":"172.21.15.79:6813/3381761360","state":["exists","up"]},{"osd":5,"uuid":"e3a56fd4-99df-4230-a51f-382af53afef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6809","nonce":862706194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6811","nonce":862706194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6815","nonce":862706194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":862706194},{"type":"v1","addr":"172.21.15.191:6813","nonce":862706194}]},"public_addr":"172.21.15.191:6809/862706194","cluster_addr":"172.21.15.191:6811/862706194","heartbeat_back_addr":"172.21.15.191:6815/862706194","heartbeat_front_addr":"172.21.15.191:6813/862706194","state":["exists","up"]},{"osd":6,"uuid":"309e6139-7f6e-4a92-bce4-e240cdc72f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6819","nonce":3661372073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6821","nonce":3661372073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6825","nonce":3661372073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3661372073},{"type":"v1","addr":"172.21.15.12:6823","nonce":3661372073}]},"public_addr":"172.21.15.12:6819/3661372073","cluster_addr":"172.21.15.12:6821/3661372073","heartbeat_back_addr":"172.21.15.12:6825/3661372073","heartbeat_front_addr":"172.21.15.12:6823/3661372073","state":["exists","up"]},{"osd":7,"uuid":"9f77cf43-cd10-4ef1-bb88-dbe4f154c353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6817","nonce":1956068856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6819","nonce":1956068856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6823","nonce":1956068856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1956068856},{"type":"v1","addr":"172.21.15.79:6821","nonce":1956068856}]},"public_addr":"172.21.15.79:6817/1956068856","cluster_addr":"172.21.15.79:6819/1956068856","heartbeat_back_addr":"172.21.15.79:6823/1956068856","heartbeat_front_addr":"172.21.15.79:6821/1956068856","state":["exists","up"]},{"osd":8,"uuid":"9ae081be-d65c-456a-81f5-8a299254b8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6817","nonce":704430703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6819","nonce":704430703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6823","nonce":704430703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":704430703},{"type":"v1","addr":"172.21.15.191:6821","nonce":704430703}]},"public_addr":"172.21.15.191:6817/704430703","cluster_addr":"172.21.15.191:6819/704430703","heartbeat_back_addr":"172.21.15.191:6823/704430703","heartbeat_front_addr":"172.21.15.191:6821/704430703","state":["exists","up"]},{"osd":9,"uuid":"f8a4b561-6262-4a05-855f-086c7d39a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6827","nonce":140672796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6829","nonce":140672796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6833","nonce":140672796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":140672796},{"type":"v1","addr":"172.21.15.12:6831","nonce":140672796}]},"public_addr":"172.21.15.12:6827/140672796","cluster_addr":"172.21.15.12:6829/140672796","heartbeat_back_addr":"172.21.15.12:6833/140672796","heartbeat_front_addr":"172.21.15.12:6831/140672796","state":["exists","up"]},{"osd":10,"uuid":"4d562fe1-e7a4-41f6-81e9-95f8745539bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6825","nonce":3620799359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6827","nonce":3620799359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6831","nonce":3620799359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":3620799359},{"type":"v1","addr":"172.21.15.79:6829","nonce":3620799359}]},"public_addr":"172.21.15.79:6825/3620799359","cluster_addr":"172.21.15.79:6827/3620799359","heartbeat_back_addr":"172.21.15.79:6831/3620799359","heartbeat_front_addr":"172.21.15.79:6829/3620799359","state":["exists","up"]},{"osd":11,"uuid":"2ccb10cd-9e17-4f3e-9d75-c55a2c9f8a58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6825","nonce":1030815303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6827","nonce":1030815303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6831","nonce":1030815303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1030815303},{"type":"v1","addr":"172.21.15.191:6829","nonce":1030815303}]},"public_addr":"172.21.15.191:6825/1030815303","cluster_addr":"172.21.15.191:6827/1030815303","heartbeat_back_addr":"172.21.15.191:6831/1030815303","heartbeat_front_addr":"172.21.15.191:6829/1030815303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:48:46.691884+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:12.018862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:49:35.774431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:10.022585+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:50:36.366913+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:01.465849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:51:35.809084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:02.356029+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:52:27.977708+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:02.715949+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:29.411097+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:53:55.611466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/2336882461":"2024-03-18T23:46:53.955890+0000","172.21.15.12:6800/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1976882257":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/501542164":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/654354506":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6801/1563339918":"2024-03-18T23:46:19.706199+0000","172.21.15.12:0/1974593432":"2024-03-18T23:46:19.706199+0000","172.21.15.12:6800/2526360059":"2024-03-18T23:46:53.955890+0000","172.21.15.12:0/1690054132":"2024-03-18T23:46:19.706199+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:07:30.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.210+0000 7f1c7e000640 1 -- 172.21.15.12:0/58420602 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1c600a2800 msgr2=0x7f1c600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:30.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.210+0000 7f1c7e000640 1 --2- 172.21.15.12:0/58420602 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1c600a2800 0x7f1c600a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c00bfd0 tx=0x7f1c6c00b390 comp rx=0 tx=0).stop 2024-03-18T00:07:30.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.210+0000 7f1c7e000640 1 -- 172.21.15.12:0/58420602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c78107350 msgr2=0x7f1c781005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:30.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.210+0000 7f1c7e000640 1 --2- 172.21.15.12:0/58420602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c78107350 0x7f1c781005e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000eb10 tx=0x7f1c7000c2d0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.211+0000 7f1c7e000640 1 -- 172.21.15.12:0/58420602 shutdown_connections 2024-03-18T00:07:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.211+0000 7f1c7e000640 1 --2- 172.21.15.12:0/58420602 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f1c600a2800 0x7f1c600a4cc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.211+0000 7f1c7e000640 1 --2- 172.21.15.12:0/58420602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c78107350 0x7f1c781005e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.211+0000 7f1c7e000640 1 --2- 172.21.15.12:0/58420602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c78104a20 0x7f1c781000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.211+0000 7f1c7e000640 1 --2- 172.21.15.12:0/58420602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c781020f0 0x7f1c78101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.211+0000 7f1c7e000640 1 -- 172.21.15.12:0/58420602 >> 172.21.15.12:0/58420602 conn(0x7f1c780fba60 msgr2=0x7f1c78102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.211+0000 7f1c7e000640 1 -- 172.21.15.12:0/58420602 shutdown_connections 2024-03-18T00:07:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.211+0000 7f1c7e000640 1 -- 172.21.15.12:0/58420602 wait complete. 2024-03-18T00:07:30.234 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T00:07:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.442+0000 7fa1b3c00640 1 -- 172.21.15.12:0/2629313084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1ac103a20 msgr2=0x7fa1ac103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.442+0000 7fa1b3c00640 1 --2- 172.21.15.12:0/2629313084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1ac103a20 0x7fa1ac103e40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0009900 tx=0x7fa1a002bc30 comp rx=0 tx=0).stop 2024-03-18T00:07:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.442+0000 7fa1b3c00640 1 -- 172.21.15.12:0/2629313084 shutdown_connections 2024-03-18T00:07:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.442+0000 7fa1b3c00640 1 --2- 172.21.15.12:0/2629313084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa1ac1055d0 0x7fa1ac10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.442+0000 7fa1b3c00640 1 --2- 172.21.15.12:0/2629313084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ac104c10 0x7fa1ac105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.442+0000 7fa1b3c00640 1 --2- 172.21.15.12:0/2629313084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1ac103a20 0x7fa1ac103e40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.442+0000 7fa1b3c00640 1 -- 172.21.15.12:0/2629313084 >> 172.21.15.12:0/2629313084 conn(0x7fa1ac0fef90 msgr2=0x7fa1ac1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.442+0000 7fa1b3c00640 1 -- 172.21.15.12:0/2629313084 shutdown_connections 2024-03-18T00:07:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.442+0000 7fa1b3c00640 1 -- 172.21.15.12:0/2629313084 wait complete. 2024-03-18T00:07:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.443+0000 7fa1b3c00640 1 Processor -- start 2024-03-18T00:07:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.444+0000 7fa1b3c00640 1 -- start start 2024-03-18T00:07:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.444+0000 7fa1b3c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1ac103a20 0x7fa1ac199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.444+0000 7fa1b3c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa1ac104c10 0x7fa1ac19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.444+0000 7fa1b3c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ac1055d0 0x7fa1ac1a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.444+0000 7fa1b3c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac10f6c0 con 0x7fa1ac1055d0 2024-03-18T00:07:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.444+0000 7fa1b3c00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac10f540 con 0x7fa1ac104c10 2024-03-18T00:07:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.444+0000 7fa1b3c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac10f840 con 0x7fa1ac103a20 2024-03-18T00:07:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.444+0000 7fa1b1200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1ac103a20 0x7fa1ac199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.444+0000 7fa1b1200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1ac103a20 0x7fa1ac199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.12:40060/0 (socket says 172.21.15.12:40060) 2024-03-18T00:07:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.444+0000 7fa1b1200640 1 -- 172.21.15.12:0/3646855046 learned_addr learned my addr 172.21.15.12:0/3646855046 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.444+0000 7fa1abe00640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa1ac104c10 0x7fa1ac19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.444+0000 7fa1b1c00640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ac1055d0 0x7fa1ac1a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.445+0000 7fa1b1200640 1 -- 172.21.15.12:0/3646855046 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa1ac104c10 msgr2=0x7fa1ac19a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.445+0000 7fa1b1200640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa1ac104c10 0x7fa1ac19a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.445+0000 7fa1b1200640 1 -- 172.21.15.12:0/3646855046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ac1055d0 msgr2=0x7fa1ac1a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.445+0000 7fa1b1200640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ac1055d0 0x7fa1ac1a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.445+0000 7fa1b1200640 1 -- 172.21.15.12:0/3646855046 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ac1a1980 con 0x7fa1ac103a20 2024-03-18T00:07:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.445+0000 7fa1b1200640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1ac103a20 0x7fa1ac199d60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0000c00 tx=0x7fa1a002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.445+0000 7fa1a9600640 1 -- 172.21.15.12:0/3646855046 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0043020 con 0x7fa1ac103a20 2024-03-18T00:07:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.445+0000 7fa1a9600640 1 -- 172.21.15.12:0/3646855046 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1a00054e0 con 0x7fa1ac103a20 2024-03-18T00:07:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.445+0000 7fa1b3c00640 1 -- 172.21.15.12:0/3646855046 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1ac1a1c10 con 0x7fa1ac103a20 2024-03-18T00:07:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.445+0000 7fa1a9600640 1 -- 172.21.15.12:0/3646855046 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0003c40 con 0x7fa1ac103a20 2024-03-18T00:07:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.446+0000 7fa1b3c00640 1 -- 172.21.15.12:0/3646855046 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1ac1a21b0 con 0x7fa1ac103a20 2024-03-18T00:07:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.447+0000 7fa1b3c00640 1 -- 172.21.15.12:0/3646855046 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1ac10ad90 con 0x7fa1ac103a20 2024-03-18T00:07:30.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.450+0000 7fa1a9600640 1 -- 172.21.15.12:0/3646855046 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fa1a0046020 con 0x7fa1ac103a20 2024-03-18T00:07:30.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.451+0000 7fa1a9600640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa1940a2880 0x7fa1940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:30.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.451+0000 7fa1a9600640 1 -- 172.21.15.12:0/3646855046 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fa1a00f0e80 con 0x7fa1ac103a20 2024-03-18T00:07:30.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.452+0000 7fa1abe00640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa1940a2880 0x7fa1940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:30.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.452+0000 7fa1abe00640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa1940a2880 0x7fa1940a4d40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa1980095f0 tx=0x7fa198005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:30.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.460+0000 7fa1a9600640 1 -- 172.21.15.12:0/3646855046 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa1a004d020 con 0x7fa1ac103a20 2024-03-18T00:07:30.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.780+0000 7fa1b3c00640 1 -- 172.21.15.12:0/3646855046 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fa1ac0623e0 con 0x7fa1ac103a20 2024-03-18T00:07:30.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.791+0000 7fa1a9600640 1 -- 172.21.15.12:0/3646855046 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v50) ==== 161+0+0 (secure 0 0 0) 0x7fa1a00bdb40 con 0x7fa1ac103a20 2024-03-18T00:07:30.792 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T00:07:30.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.797+0000 7fa1b3c00640 1 -- 172.21.15.12:0/3646855046 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa1940a2880 msgr2=0x7fa1940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:30.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.797+0000 7fa1b3c00640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa1940a2880 0x7fa1940a4d40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa1980095f0 tx=0x7fa198005f90 comp rx=0 tx=0).stop 2024-03-18T00:07:30.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.797+0000 7fa1b3c00640 1 -- 172.21.15.12:0/3646855046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1ac103a20 msgr2=0x7fa1ac199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:30.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.797+0000 7fa1b3c00640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1ac103a20 0x7fa1ac199d60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0000c00 tx=0x7fa1a002c650 comp rx=0 tx=0).stop 2024-03-18T00:07:30.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.797+0000 7fa1b3c00640 1 -- 172.21.15.12:0/3646855046 shutdown_connections 2024-03-18T00:07:30.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.797+0000 7fa1b3c00640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fa1940a2880 0x7fa1940a4d40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.797+0000 7fa1b3c00640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1ac1055d0 0x7fa1ac1a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.797+0000 7fa1b3c00640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa1ac104c10 0x7fa1ac19a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.797+0000 7fa1b3c00640 1 --2- 172.21.15.12:0/3646855046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1ac103a20 0x7fa1ac199d60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:30.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.797+0000 7fa1b3c00640 1 -- 172.21.15.12:0/3646855046 >> 172.21.15.12:0/3646855046 conn(0x7fa1ac0fef90 msgr2=0x7fa1ac107f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:30.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.798+0000 7fa1b3c00640 1 -- 172.21.15.12:0/3646855046 shutdown_connections 2024-03-18T00:07:30.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:30.798+0000 7fa1b3c00640 1 -- 172.21.15.12:0/3646855046 wait complete. 2024-03-18T00:07:30.826 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T00:07:30.894 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 18 00:07:30 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-a[70469]: ignoring --setuser ceph since I am not root 2024-03-18T00:07:30.894 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 18 00:07:30 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-a[70469]: ignoring --setgroup ceph since I am not root 2024-03-18T00:07:30.894 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 18 00:07:30 smithi012 ceph-mds[70473]: -- 172.21.15.12:0/867638752 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5574df0e3a40 con 0x5574df0b6800 2024-03-18T00:07:30.894 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 18 00:07:30 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-a[70469]: starting mds.a at 2024-03-18T00:07:30.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:30 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1760404770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:07:30.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:30 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/58420602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:07:30.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:30 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a[31537]: 2024-03-18T00:07:30.782+0000 7f63a5c00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T00:07:30.895 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 18 00:07:30 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d[69469]: ignoring --setuser ceph since I am not root 2024-03-18T00:07:30.895 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 18 00:07:30 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d[69469]: ignoring --setgroup ceph since I am not root 2024-03-18T00:07:30.895 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 18 00:07:30 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d[69469]: starting mds.d at 2024-03-18T00:07:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.025+0000 7fd68da00640 1 -- 172.21.15.12:0/3393458303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd688104c10 msgr2=0x7fd688105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.025+0000 7fd68da00640 1 --2- 172.21.15.12:0/3393458303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd688104c10 0x7fd688105090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd674009900 tx=0x7fd67402bc30 comp rx=0 tx=0).stop 2024-03-18T00:07:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.025+0000 7fd68da00640 1 -- 172.21.15.12:0/3393458303 shutdown_connections 2024-03-18T00:07:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.025+0000 7fd68da00640 1 --2- 172.21.15.12:0/3393458303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6881055d0 0x7fd68810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.025+0000 7fd68da00640 1 --2- 172.21.15.12:0/3393458303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd688104c10 0x7fd688105090 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.025+0000 7fd68da00640 1 --2- 172.21.15.12:0/3393458303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd688103a20 0x7fd688103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.025+0000 7fd68da00640 1 -- 172.21.15.12:0/3393458303 >> 172.21.15.12:0/3393458303 conn(0x7fd6880fef90 msgr2=0x7fd6881013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:31.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.026+0000 7fd68da00640 1 -- 172.21.15.12:0/3393458303 shutdown_connections 2024-03-18T00:07:31.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.026+0000 7fd68da00640 1 -- 172.21.15.12:0/3393458303 wait complete. 2024-03-18T00:07:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.027+0000 7fd68da00640 1 Processor -- start 2024-03-18T00:07:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.027+0000 7fd68da00640 1 -- start start 2024-03-18T00:07:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.027+0000 7fd68da00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd688103a20 0x7fd688199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.027+0000 7fd68da00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd688104c10 0x7fd68819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.027+0000 7fd68da00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6881055d0 0x7fd6881a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.027+0000 7fd68da00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd68810f6d0 con 0x7fd688104c10 2024-03-18T00:07:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.027+0000 7fd68da00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd68810f550 con 0x7fd688103a20 2024-03-18T00:07:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.027+0000 7fd68da00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd68810f850 con 0x7fd6881055d0 2024-03-18T00:07:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.027+0000 7fd686a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd688103a20 0x7fd688199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.027+0000 7fd686000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd688104c10 0x7fd68819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.028+0000 7fd686a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd688103a20 0x7fd688199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60176/0 (socket says 172.21.15.12:60176) 2024-03-18T00:07:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.028+0000 7fd686a00640 1 -- 172.21.15.12:0/2504168779 learned_addr learned my addr 172.21.15.12:0/2504168779 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.028+0000 7fd687400640 1 --2- 172.21.15.12:0/2504168779 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6881055d0 0x7fd6881a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.028+0000 7fd686a00640 1 -- 172.21.15.12:0/2504168779 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6881055d0 msgr2=0x7fd6881a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.028+0000 7fd686a00640 1 --2- 172.21.15.12:0/2504168779 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6881055d0 0x7fd6881a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.028+0000 7fd686a00640 1 -- 172.21.15.12:0/2504168779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd688104c10 msgr2=0x7fd68819a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.028+0000 7fd686a00640 1 --2- 172.21.15.12:0/2504168779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd688104c10 0x7fd68819a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:31.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.028+0000 7fd686a00640 1 -- 172.21.15.12:0/2504168779 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6881a1940 con 0x7fd688103a20 2024-03-18T00:07:31.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.028+0000 7fd686a00640 1 --2- 172.21.15.12:0/2504168779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd688103a20 0x7fd688199d30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd67800d770 tx=0x7fd67800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:31.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.028+0000 7fd67f400640 1 -- 172.21.15.12:0/2504168779 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678017070 con 0x7fd688103a20 2024-03-18T00:07:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.029+0000 7fd67f400640 1 -- 172.21.15.12:0/2504168779 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd678009c30 con 0x7fd688103a20 2024-03-18T00:07:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.029+0000 7fd67f400640 1 -- 172.21.15.12:0/2504168779 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678003dd0 con 0x7fd688103a20 2024-03-18T00:07:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.029+0000 7fd68da00640 1 -- 172.21.15.12:0/2504168779 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6881a1c30 con 0x7fd688103a20 2024-03-18T00:07:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.029+0000 7fd68da00640 1 -- 172.21.15.12:0/2504168779 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6881a2130 con 0x7fd688103a20 2024-03-18T00:07:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.030+0000 7fd68da00640 1 -- 172.21.15.12:0/2504168779 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd68810ad40 con 0x7fd688103a20 2024-03-18T00:07:31.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.032+0000 7fd67f400640 1 -- 172.21.15.12:0/2504168779 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fd678003f70 con 0x7fd688103a20 2024-03-18T00:07:31.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.033+0000 7fd67f400640 1 --2- 172.21.15.12:0/2504168779 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd66c0a2880 0x7fd66c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:31.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.033+0000 7fd67f400640 1 -- 172.21.15.12:0/2504168779 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9551+0+0 (secure 0 0 0) 0x7fd6780cec80 con 0x7fd688103a20 2024-03-18T00:07:31.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.033+0000 7fd686000640 1 --2- 172.21.15.12:0/2504168779 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd66c0a2880 0x7fd66c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:31.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.034+0000 7fd686000640 1 --2- 172.21.15.12:0/2504168779 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd66c0a2880 0x7fd66c0a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd67402c240 tx=0x7fd67402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:31.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.035+0000 7fd67f400640 1 -- 172.21.15.12:0/2504168779 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd678009de0 con 0x7fd688103a20 2024-03-18T00:07:31.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1760404770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:07:31.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/58420602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:07:31.046 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-k[65337]: ignoring --setuser ceph since I am not root 2024-03-18T00:07:31.046 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-k[65337]: ignoring --setgroup ceph since I am not root 2024-03-18T00:07:31.046 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-mds[65342]: -- 172.21.15.79:0/2256875343 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564d367cfa40 con 0x564d367a3000 2024-03-18T00:07:31.046 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-k[65337]: starting mds.k at 2024-03-18T00:07:31.046 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-e[65654]: ignoring --setuser ceph since I am not root 2024-03-18T00:07:31.046 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-e[65654]: ignoring --setgroup ceph since I am not root 2024-03-18T00:07:31.047 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-mds[65658]: -- 172.21.15.79:0/2798762924 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5573876c7a40 con 0x55738769a800 2024-03-18T00:07:31.047 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-e[65654]: starting mds.e at 2024-03-18T00:07:31.047 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-h[65960]: ignoring --setuser ceph since I am not root 2024-03-18T00:07:31.047 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-h[65960]: ignoring --setgroup ceph since I am not root 2024-03-18T00:07:31.047 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-mds[65964]: -- 172.21.15.79:0/3581289512 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e39a75fa40 con 0x55e39a732800 2024-03-18T00:07:31.047 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-h[65960]: starting mds.h at 2024-03-18T00:07:31.047 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b[66285]: ignoring --setuser ceph since I am not root 2024-03-18T00:07:31.048 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b[66285]: ignoring --setgroup ceph since I am not root 2024-03-18T00:07:31.048 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-mds[66289]: -- 172.21.15.79:0/1955406150 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560d007b9a40 con 0x560d0078d000 2024-03-18T00:07:31.048 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 18 00:07:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b[66285]: starting mds.b at 2024-03-18T00:07:31.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1760404770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:07:31.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/58420602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:07:31.091 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-f[67197]: ignoring --setuser ceph since I am not root 2024-03-18T00:07:31.091 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-f[67197]: ignoring --setgroup ceph since I am not root 2024-03-18T00:07:31.091 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-mds[67201]: -- 172.21.15.191:0/3248197530 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b067a6ba40 con 0x55b067a3e800 2024-03-18T00:07:31.092 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-f[67197]: starting mds.f at 2024-03-18T00:07:31.092 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-i[67817]: ignoring --setuser ceph since I am not root 2024-03-18T00:07:31.092 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-i[67817]: ignoring --setgroup ceph since I am not root 2024-03-18T00:07:31.092 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-mds[67821]: -- 172.21.15.191:0/3270648664 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56212bfafa40 con 0x56212bf82800 2024-03-18T00:07:31.092 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-i[67817]: starting mds.i at 2024-03-18T00:07:31.092 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-l[66890]: ignoring --setuser ceph since I am not root 2024-03-18T00:07:31.092 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-l[66890]: ignoring --setgroup ceph since I am not root 2024-03-18T00:07:31.093 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-mds[66894]: -- 172.21.15.191:0/894526388 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dfedcb3a40 con 0x55dfedc86800 2024-03-18T00:07:31.093 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-l[66890]: starting mds.l at 2024-03-18T00:07:31.093 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 00:07:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-c[67504]: starting mds.c at 2024-03-18T00:07:31.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.297+0000 7fd68da00640 1 -- 172.21.15.12:0/2504168779 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd68802d460 con 0x7fd688103a20 2024-03-18T00:07:31.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.792+0000 7fd67f400640 1 -- 172.21.15.12:0/2504168779 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v51) ==== 104+0+0 (secure 0 0 0) 0x7fd67809b1d0 con 0x7fd688103a20 2024-03-18T00:07:31.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.794+0000 7fd68da00640 1 -- 172.21.15.12:0/2504168779 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd66c0a2880 msgr2=0x7fd66c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.794+0000 7fd68da00640 1 --2- 172.21.15.12:0/2504168779 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd66c0a2880 0x7fd66c0a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd67402c240 tx=0x7fd67402df90 comp rx=0 tx=0).stop 2024-03-18T00:07:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.794+0000 7fd68da00640 1 -- 172.21.15.12:0/2504168779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd688103a20 msgr2=0x7fd688199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.794+0000 7fd68da00640 1 --2- 172.21.15.12:0/2504168779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd688103a20 0x7fd688199d30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd67800d770 tx=0x7fd67800dc30 comp rx=0 tx=0).stop 2024-03-18T00:07:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.795+0000 7fd68da00640 1 -- 172.21.15.12:0/2504168779 shutdown_connections 2024-03-18T00:07:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.795+0000 7fd68da00640 1 --2- 172.21.15.12:0/2504168779 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fd66c0a2880 0x7fd66c0a4d40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.795+0000 7fd68da00640 1 --2- 172.21.15.12:0/2504168779 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6881055d0 0x7fd6881a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:31.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.795+0000 7fd68da00640 1 --2- 172.21.15.12:0/2504168779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd688104c10 0x7fd68819a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:31.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.795+0000 7fd68da00640 1 --2- 172.21.15.12:0/2504168779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd688103a20 0x7fd688199d30 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:31.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.795+0000 7fd68da00640 1 -- 172.21.15.12:0/2504168779 >> 172.21.15.12:0/2504168779 conn(0x7fd6880fef90 msgr2=0x7fd688107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:31.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.795+0000 7fd68da00640 1 -- 172.21.15.12:0/2504168779 shutdown_connections 2024-03-18T00:07:31.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:31.795+0000 7fd68da00640 1 -- 172.21.15.12:0/2504168779 wait complete. 2024-03-18T00:07:31.819 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T00:07:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.017+0000 7f3865c00640 1 -- 172.21.15.12:0/4034531932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3860103270 msgr2=0x7f3860109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.017+0000 7f3865c00640 1 --2- 172.21.15.12:0/4034531932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3860103270 0x7f3860109b00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3858009ca0 tx=0x7f385802c100 comp rx=0 tx=0).stop 2024-03-18T00:07:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.017+0000 7f3865c00640 1 -- 172.21.15.12:0/4034531932 shutdown_connections 2024-03-18T00:07:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.017+0000 7f3865c00640 1 --2- 172.21.15.12:0/4034531932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3860103270 0x7f3860109b00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.017+0000 7f3865c00640 1 --2- 172.21.15.12:0/4034531932 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38601028b0 0x7f3860102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.017+0000 7f3865c00640 1 --2- 172.21.15.12:0/4034531932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38601016c0 0x7f3860101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.017+0000 7f3865c00640 1 -- 172.21.15.12:0/4034531932 >> 172.21.15.12:0/4034531932 conn(0x7f38600fcc50 msgr2=0x7f38600ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.017+0000 7f3865c00640 1 -- 172.21.15.12:0/4034531932 shutdown_connections 2024-03-18T00:07:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.017+0000 7f3865c00640 1 -- 172.21.15.12:0/4034531932 wait complete. 2024-03-18T00:07:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.018+0000 7f3865c00640 1 Processor -- start 2024-03-18T00:07:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.018+0000 7f3865c00640 1 -- start start 2024-03-18T00:07:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.018+0000 7f3865c00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38601016c0 0x7f3860199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.018+0000 7f3865c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38601028b0 0x7f386019a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.018+0000 7f3865c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3860103270 0x7f38601a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.018+0000 7f3865c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f386010d320 con 0x7f3860103270 2024-03-18T00:07:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.018+0000 7f3865c00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f386010d1a0 con 0x7f38601016c0 2024-03-18T00:07:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.018+0000 7f3865c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f386010d4a0 con 0x7f38601028b0 2024-03-18T00:07:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.019+0000 7f385fe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38601016c0 0x7f3860199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.019+0000 7f3865200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3860103270 0x7f38601a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.019+0000 7f385fe00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38601016c0 0x7f3860199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60202/0 (socket says 172.21.15.12:60202) 2024-03-18T00:07:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.019+0000 7f385fe00640 1 -- 172.21.15.12:0/2256855115 learned_addr learned my addr 172.21.15.12:0/2256855115 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.019+0000 7f3857e00640 1 --2- 172.21.15.12:0/2256855115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38601028b0 0x7f386019a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.019+0000 7f385fe00640 1 -- 172.21.15.12:0/2256855115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38601028b0 msgr2=0x7f386019a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.019+0000 7f385fe00640 1 --2- 172.21.15.12:0/2256855115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38601028b0 0x7f386019a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.019+0000 7f385fe00640 1 -- 172.21.15.12:0/2256855115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3860103270 msgr2=0x7f38601a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.019+0000 7f385fe00640 1 --2- 172.21.15.12:0/2256855115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3860103270 0x7f38601a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.019+0000 7f385fe00640 1 -- 172.21.15.12:0/2256855115 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38601a1770 con 0x7f38601016c0 2024-03-18T00:07:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.019+0000 7f385fe00640 1 --2- 172.21.15.12:0/2256855115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38601016c0 0x7f3860199b50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f385000e9c0 tx=0x7f385000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.020+0000 7f385d600640 1 -- 172.21.15.12:0/2256855115 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385000cde0 con 0x7f38601016c0 2024-03-18T00:07:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.020+0000 7f385d600640 1 -- 172.21.15.12:0/2256855115 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3850010910 con 0x7f38601016c0 2024-03-18T00:07:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.020+0000 7f385d600640 1 -- 172.21.15.12:0/2256855115 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3850018a00 con 0x7f38601016c0 2024-03-18T00:07:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.020+0000 7f3865c00640 1 -- 172.21.15.12:0/2256855115 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38601a1a60 con 0x7f38601016c0 2024-03-18T00:07:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.020+0000 7f3865c00640 1 -- 172.21.15.12:0/2256855115 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38601a1fb0 con 0x7f38601016c0 2024-03-18T00:07:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.021+0000 7f3865c00640 1 -- 172.21.15.12:0/2256855115 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3860068940 con 0x7f38601016c0 2024-03-18T00:07:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.023+0000 7f385d600640 1 -- 172.21.15.12:0/2256855115 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f385001f020 con 0x7f38601016c0 2024-03-18T00:07:32.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.024+0000 7f385d600640 1 --2- 172.21.15.12:0/2256855115 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f38440a2800 0x7f38440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:32.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.024+0000 7f385d600640 1 -- 172.21.15.12:0/2256855115 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9551+0+0 (secure 0 0 0) 0x7f38500cf790 con 0x7f38601016c0 2024-03-18T00:07:32.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.024+0000 7f3857e00640 1 --2- 172.21.15.12:0/2256855115 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f38440a2800 0x7f38440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:32.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.025+0000 7f3857e00640 1 --2- 172.21.15.12:0/2256855115 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f38440a2800 0x7f38440a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f386019b060 tx=0x7f3848005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:32.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.027+0000 7f385d600640 1 -- 172.21.15.12:0/2256855115 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f385009c2b0 con 0x7f38601016c0 2024-03-18T00:07:32.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:31 smithi079 ceph-mon[36241]: pgmap v702: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T00:07:32.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:31 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/3646855046' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T00:07:32.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:31 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T00:07:32.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:31 smithi079 ceph-mon[36241]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T00:07:32.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:31 smithi079 ceph-mon[36241]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T00:07:32.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:31 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T00:07:32.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:31 smithi079 ceph-mon[36241]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T00:07:32.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:31 smithi079 ceph-mon[36241]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T00:07:32.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:31 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2504168779' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T00:07:32.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:31 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T00:07:32.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:31 smithi191 ceph-mon[36708]: pgmap v702: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T00:07:32.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:31 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/3646855046' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T00:07:32.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:31 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T00:07:32.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:31 smithi191 ceph-mon[36708]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T00:07:32.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:31 smithi191 ceph-mon[36708]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T00:07:32.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:31 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T00:07:32.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:31 smithi191 ceph-mon[36708]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T00:07:32.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:31 smithi191 ceph-mon[36708]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T00:07:32.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:31 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2504168779' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T00:07:32.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:31 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T00:07:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:31 smithi012 ceph-mon[31541]: pgmap v702: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T00:07:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:31 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/3646855046' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T00:07:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:31 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T00:07:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:31 smithi012 ceph-mon[31541]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T00:07:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:31 smithi012 ceph-mon[31541]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T00:07:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:31 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T00:07:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:31 smithi012 ceph-mon[31541]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T00:07:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:31 smithi012 ceph-mon[31541]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T00:07:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:31 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2504168779' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T00:07:32.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:31 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T00:07:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.224+0000 7f3865c00640 1 -- 172.21.15.12:0/2256855115 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f38601a21e0 con 0x7f38601016c0 2024-03-18T00:07:32.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.797+0000 7f385d600640 1 -- 172.21.15.12:0/2256855115 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v96) ==== 181+0+0 (secure 0 0 0) 0x7f385009bbb0 con 0x7f38601016c0 2024-03-18T00:07:32.800 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-03-18T00:07:32.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.804+0000 7f3865c00640 1 -- 172.21.15.12:0/2256855115 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f38440a2800 msgr2=0x7f38440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:32.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.804+0000 7f3865c00640 1 --2- 172.21.15.12:0/2256855115 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f38440a2800 0x7f38440a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f386019b060 tx=0x7f3848005f90 comp rx=0 tx=0).stop 2024-03-18T00:07:32.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.804+0000 7f3865c00640 1 -- 172.21.15.12:0/2256855115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38601016c0 msgr2=0x7f3860199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:32.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.804+0000 7f3865c00640 1 --2- 172.21.15.12:0/2256855115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38601016c0 0x7f3860199b50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f385000e9c0 tx=0x7f385000ee80 comp rx=0 tx=0).stop 2024-03-18T00:07:32.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.804+0000 7f3865c00640 1 -- 172.21.15.12:0/2256855115 shutdown_connections 2024-03-18T00:07:32.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.804+0000 7f3865c00640 1 --2- 172.21.15.12:0/2256855115 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7f38440a2800 0x7f38440a4cc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:32.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.804+0000 7f3865c00640 1 --2- 172.21.15.12:0/2256855115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3860103270 0x7f38601a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:32.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.804+0000 7f3865c00640 1 --2- 172.21.15.12:0/2256855115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38601028b0 0x7f386019a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.804+0000 7f3865c00640 1 --2- 172.21.15.12:0/2256855115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38601016c0 0x7f3860199b50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.804+0000 7f3865c00640 1 -- 172.21.15.12:0/2256855115 >> 172.21.15.12:0/2256855115 conn(0x7f38600fcc50 msgr2=0x7f3860073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.805+0000 7f3865c00640 1 -- 172.21.15.12:0/2256855115 shutdown_connections 2024-03-18T00:07:32.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:32.805+0000 7f3865c00640 1 -- 172.21.15.12:0/2256855115 wait complete. 2024-03-18T00:07:32.828 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T00:07:33.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.039+0000 7fbee3e00640 1 -- 172.21.15.12:0/3834248351 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee4105490 msgr2=0x7fbee410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:33.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.039+0000 7fbee3e00640 1 --2- 172.21.15.12:0/3834248351 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee4105490 0x7fbee410bd20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbedc009be0 tx=0x7fbedc02bef0 comp rx=0 tx=0).stop 2024-03-18T00:07:33.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.039+0000 7fbee3e00640 1 -- 172.21.15.12:0/3834248351 shutdown_connections 2024-03-18T00:07:33.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.039+0000 7fbee3e00640 1 --2- 172.21.15.12:0/3834248351 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee4105490 0x7fbee410bd20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:33.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.039+0000 7fbee3e00640 1 --2- 172.21.15.12:0/3834248351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbee4104ad0 0x7fbee4104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:33.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.039+0000 7fbee3e00640 1 --2- 172.21.15.12:0/3834248351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbee41038e0 0x7fbee4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:33.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.039+0000 7fbee3e00640 1 -- 172.21.15.12:0/3834248351 >> 172.21.15.12:0/3834248351 conn(0x7fbee40fee50 msgr2=0x7fbee4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:33.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.039+0000 7fbee3e00640 1 -- 172.21.15.12:0/3834248351 shutdown_connections 2024-03-18T00:07:33.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.039+0000 7fbee3e00640 1 -- 172.21.15.12:0/3834248351 wait complete. 2024-03-18T00:07:33.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.040+0000 7fbee3e00640 1 Processor -- start 2024-03-18T00:07:33.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.040+0000 7fbee3e00640 1 -- start start 2024-03-18T00:07:33.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee3e00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee41038e0 0x7fbee4199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbee4104ad0 0x7fbee419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbee4105490 0x7fbee41a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee3e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbee410f540 con 0x7fbee4105490 2024-03-18T00:07:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee3e00640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbee410f3c0 con 0x7fbee41038e0 2024-03-18T00:07:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee3e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbee410f6c0 con 0x7fbee4104ad0 2024-03-18T00:07:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbee4105490 0x7fbee41a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee2a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee41038e0 0x7fbee4199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee2000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbee4104ad0 0x7fbee419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee2a00640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee41038e0 0x7fbee4199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:60220/0 (socket says 172.21.15.12:60220) 2024-03-18T00:07:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee2a00640 1 -- 172.21.15.12:0/1263337311 learned_addr learned my addr 172.21.15.12:0/1263337311 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee2a00640 1 -- 172.21.15.12:0/1263337311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbee4104ad0 msgr2=0x7fbee419a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee2a00640 1 --2- 172.21.15.12:0/1263337311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbee4104ad0 0x7fbee419a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee2a00640 1 -- 172.21.15.12:0/1263337311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbee4105490 msgr2=0x7fbee41a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee2a00640 1 --2- 172.21.15.12:0/1263337311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbee4105490 0x7fbee41a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.041+0000 7fbee2a00640 1 -- 172.21.15.12:0/1263337311 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee41a17a0 con 0x7fbee41038e0 2024-03-18T00:07:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.042+0000 7fbee2a00640 1 --2- 172.21.15.12:0/1263337311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee41038e0 0x7fbee4199b80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbed400e9f0 tx=0x7fbed400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.042+0000 7fbedb400640 1 -- 172.21.15.12:0/1263337311 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed400ccc0 con 0x7fbee41038e0 2024-03-18T00:07:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.042+0000 7fbedb400640 1 -- 172.21.15.12:0/1263337311 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbed40035f0 con 0x7fbee41038e0 2024-03-18T00:07:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.042+0000 7fbee3e00640 1 -- 172.21.15.12:0/1263337311 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee41a1a90 con 0x7fbee41038e0 2024-03-18T00:07:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.042+0000 7fbedb400640 1 -- 172.21.15.12:0/1263337311 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed400ce60 con 0x7fbee41038e0 2024-03-18T00:07:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.042+0000 7fbee3e00640 1 -- 172.21.15.12:0/1263337311 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee41a2030 con 0x7fbee41038e0 2024-03-18T00:07:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.043+0000 7fbee3e00640 1 -- 172.21.15.12:0/1263337311 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee410ac80 con 0x7fbee41038e0 2024-03-18T00:07:33.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T00:07:33.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: Cluster is now healthy 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.191:6834/556708771,v1:172.21.15.191:6835/556708771] up:boot 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.79:6832/3605882398,v1:172.21.15.79:6835/3605882398] up:boot 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.191:6836/1849185248,v1:172.21.15.191:6837/1849185248] up:boot 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.79:6836/3197026136,v1:172.21.15.79:6837/3197026136] up:boot 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.191:6838/1409592088,v1:172.21.15.191:6839/1409592088] up:boot 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.79:6838/2790045249,v1:172.21.15.79:6839/2790045249] up:boot 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.191:6832/2022811792,v1:172.21.15.191:6833/2022811792] up:boot 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.79:6833/3012707741,v1:172.21.15.79:6834/3012707741] up:boot 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.12:6836/2095578804,v1:172.21.15.12:6837/2095578804] up:boot 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: mds.? [v2:172.21.15.12:6840/787577664,v1:172.21.15.12:6841/787577664] up:boot 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: fsmap 12 up:standby 2024-03-18T00:07:33.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T00:07:33.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T00:07:33.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T00:07:33.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T00:07:33.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T00:07:33.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T00:07:33.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T00:07:33.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T00:07:33.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T00:07:33.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T00:07:33.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/2256855115' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:33.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:32 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:33.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.045+0000 7fbedb400640 1 -- 172.21.15.12:0/1263337311 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fbed40037a0 con 0x7fbee41038e0 2024-03-18T00:07:33.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.047+0000 7fbedb400640 1 --2- 172.21.15.12:0/1263337311 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbec80a2880 0x7fbec80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:33.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.047+0000 7fbedb400640 1 -- 172.21.15.12:0/1263337311 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8997+0+0 (secure 0 0 0) 0x7fbed40cefd0 con 0x7fbee41038e0 2024-03-18T00:07:33.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.047+0000 7fbee2000640 1 --2- 172.21.15.12:0/1263337311 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbec80a2880 0x7fbec80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:33.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.048+0000 7fbee2000640 1 --2- 172.21.15.12:0/1263337311 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbec80a2880 0x7fbec80a4d40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbee419b090 tx=0x7fbed0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.049+0000 7fbedb400640 1 -- 172.21.15.12:0/1263337311 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbed409bd10 con 0x7fbee41038e0 2024-03-18T00:07:33.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T00:07:33.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T00:07:33.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: Cluster is now healthy 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.191:6834/556708771,v1:172.21.15.191:6835/556708771] up:boot 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.79:6832/3605882398,v1:172.21.15.79:6835/3605882398] up:boot 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.191:6836/1849185248,v1:172.21.15.191:6837/1849185248] up:boot 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.79:6836/3197026136,v1:172.21.15.79:6837/3197026136] up:boot 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.191:6838/1409592088,v1:172.21.15.191:6839/1409592088] up:boot 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.79:6838/2790045249,v1:172.21.15.79:6839/2790045249] up:boot 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.191:6832/2022811792,v1:172.21.15.191:6833/2022811792] up:boot 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.79:6833/3012707741,v1:172.21.15.79:6834/3012707741] up:boot 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.12:6836/2095578804,v1:172.21.15.12:6837/2095578804] up:boot 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: mds.? [v2:172.21.15.12:6840/787577664,v1:172.21.15.12:6841/787577664] up:boot 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: fsmap 12 up:standby 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T00:07:33.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T00:07:33.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T00:07:33.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T00:07:33.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T00:07:33.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T00:07:33.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T00:07:33.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T00:07:33.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T00:07:33.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T00:07:33.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/2256855115' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:33.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:32 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:33.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T00:07:33.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: Cluster is now healthy 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.191:6834/556708771,v1:172.21.15.191:6835/556708771] up:boot 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.79:6832/3605882398,v1:172.21.15.79:6835/3605882398] up:boot 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.191:6836/1849185248,v1:172.21.15.191:6837/1849185248] up:boot 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.79:6836/3197026136,v1:172.21.15.79:6837/3197026136] up:boot 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.191:6838/1409592088,v1:172.21.15.191:6839/1409592088] up:boot 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.79:6838/2790045249,v1:172.21.15.79:6839/2790045249] up:boot 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.191:6832/2022811792,v1:172.21.15.191:6833/2022811792] up:boot 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.79:6833/3012707741,v1:172.21.15.79:6834/3012707741] up:boot 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.12:6836/2095578804,v1:172.21.15.12:6837/2095578804] up:boot 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: mds.? [v2:172.21.15.12:6840/787577664,v1:172.21.15.12:6841/787577664] up:boot 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: fsmap 12 up:standby 2024-03-18T00:07:33.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T00:07:33.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T00:07:33.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T00:07:33.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T00:07:33.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T00:07:33.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T00:07:33.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T00:07:33.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T00:07:33.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T00:07:33.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: from='mgr.14152 172.21.15.12:0/3459338250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T00:07:33.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/2256855115' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:33.171 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:32 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:33.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.253+0000 7fbee3e00640 1 -- 172.21.15.12:0/1263337311 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbee402d1b0 con 0x7fbee41038e0 2024-03-18T00:07:33.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.811+0000 7fbedb400640 1 -- 172.21.15.12:0/1263337311 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v97) ==== 169+0+0 (secure 0 0 0) 0x7fbed409b610 con 0x7fbee41038e0 2024-03-18T00:07:33.814 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-03-18T00:07:33.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.817+0000 7fbee3e00640 1 -- 172.21.15.12:0/1263337311 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbec80a2880 msgr2=0x7fbec80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:33.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.817+0000 7fbee3e00640 1 --2- 172.21.15.12:0/1263337311 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbec80a2880 0x7fbec80a4d40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbee419b090 tx=0x7fbed0005f90 comp rx=0 tx=0).stop 2024-03-18T00:07:33.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.817+0000 7fbee3e00640 1 -- 172.21.15.12:0/1263337311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee41038e0 msgr2=0x7fbee4199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:33.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.817+0000 7fbee3e00640 1 --2- 172.21.15.12:0/1263337311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee41038e0 0x7fbee4199b80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbed400e9f0 tx=0x7fbed400eeb0 comp rx=0 tx=0).stop 2024-03-18T00:07:33.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.818+0000 7fbee3e00640 1 -- 172.21.15.12:0/1263337311 shutdown_connections 2024-03-18T00:07:33.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.818+0000 7fbee3e00640 1 --2- 172.21.15.12:0/1263337311 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7fbec80a2880 0x7fbec80a4d40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:33.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.818+0000 7fbee3e00640 1 --2- 172.21.15.12:0/1263337311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbee4105490 0x7fbee41a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:33.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.818+0000 7fbee3e00640 1 --2- 172.21.15.12:0/1263337311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbee4104ad0 0x7fbee419a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:33.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.818+0000 7fbee3e00640 1 --2- 172.21.15.12:0/1263337311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee41038e0 0x7fbee4199b80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:33.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.818+0000 7fbee3e00640 1 -- 172.21.15.12:0/1263337311 >> 172.21.15.12:0/1263337311 conn(0x7fbee40fee50 msgr2=0x7fbee4107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:33.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.818+0000 7fbee3e00640 1 -- 172.21.15.12:0/1263337311 shutdown_connections 2024-03-18T00:07:33.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:33.820+0000 7fbee3e00640 1 -- 172.21.15.12:0/1263337311 wait complete. 2024-03-18T00:07:33.852 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-18T00:07:34.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:33 smithi079 ceph-mon[36241]: pgmap v704: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-18T00:07:34.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:33 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T00:07:34.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:33 smithi079 ceph-mon[36241]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T00:07:34.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:33 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1263337311' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:34.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:33 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:34.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.077+0000 7faf6a600640 1 -- 172.21.15.12:0/4028259871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf641087e0 msgr2=0x7faf64108c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:34.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.077+0000 7faf6a600640 1 --2- 172.21.15.12:0/4028259871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf641087e0 0x7faf64108c00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009ca0 tx=0x7faf5c02c0f0 comp rx=0 tx=0).stop 2024-03-18T00:07:34.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.077+0000 7faf6a600640 1 -- 172.21.15.12:0/4028259871 shutdown_connections 2024-03-18T00:07:34.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.077+0000 7faf6a600640 1 --2- 172.21.15.12:0/4028259871 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf64110060 0x7faf64112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:34.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.077+0000 7faf6a600640 1 --2- 172.21.15.12:0/4028259871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf64109140 0x7faf6410fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:34.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.077+0000 7faf6a600640 1 --2- 172.21.15.12:0/4028259871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf641087e0 0x7faf64108c00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:34.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.077+0000 7faf6a600640 1 -- 172.21.15.12:0/4028259871 >> 172.21.15.12:0/4028259871 conn(0x7faf6406e640 msgr2=0x7faf64070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:34.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.078+0000 7faf6a600640 1 -- 172.21.15.12:0/4028259871 shutdown_connections 2024-03-18T00:07:34.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.078+0000 7faf6a600640 1 -- 172.21.15.12:0/4028259871 wait complete. 2024-03-18T00:07:34.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.080+0000 7faf6a600640 1 Processor -- start 2024-03-18T00:07:34.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.080+0000 7faf6a600640 1 -- start start 2024-03-18T00:07:34.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.080+0000 7faf6a600640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf641087e0 0x7faf641a25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:34.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.080+0000 7faf6a600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf64109140 0x7faf641a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:34.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.080+0000 7faf6a600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64110060 0x7faf641a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:34.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.080+0000 7faf6a600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faf64117e90 con 0x7faf64109140 2024-03-18T00:07:34.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.080+0000 7faf6a600640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7faf64117d10 con 0x7faf641087e0 2024-03-18T00:07:34.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.080+0000 7faf6a600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faf64118010 con 0x7faf64110060 2024-03-18T00:07:34.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.081+0000 7faf63400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf641087e0 0x7faf641a25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:34.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.081+0000 7faf62a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf64109140 0x7faf641a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:34.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.081+0000 7faf63400640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf641087e0 0x7faf641a25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.12:47154/0 (socket says 172.21.15.12:47154) 2024-03-18T00:07:34.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.081+0000 7faf63400640 1 -- 172.21.15.12:0/1385157282 learned_addr learned my addr 172.21.15.12:0/1385157282 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-18T00:07:34.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.081+0000 7faf63e00640 1 --2- 172.21.15.12:0/1385157282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64110060 0x7faf641a9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:34.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.081+0000 7faf63400640 1 -- 172.21.15.12:0/1385157282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64110060 msgr2=0x7faf641a9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:34.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.081+0000 7faf63400640 1 --2- 172.21.15.12:0/1385157282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64110060 0x7faf641a9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:34.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.081+0000 7faf63400640 1 -- 172.21.15.12:0/1385157282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf64109140 msgr2=0x7faf641a2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:34.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.081+0000 7faf63400640 1 --2- 172.21.15.12:0/1385157282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf64109140 0x7faf641a2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:34.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.081+0000 7faf63400640 1 -- 172.21.15.12:0/1385157282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf641aa1e0 con 0x7faf641087e0 2024-03-18T00:07:34.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.082+0000 7faf63400640 1 --2- 172.21.15.12:0/1385157282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf641087e0 0x7faf641a25c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7faf5c005e30 tx=0x7faf5c032f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:34.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.082+0000 7faf5be00640 1 -- 172.21.15.12:0/1385157282 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c039040 con 0x7faf641087e0 2024-03-18T00:07:34.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.082+0000 7faf5be00640 1 -- 172.21.15.12:0/1385157282 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf5c044070 con 0x7faf641087e0 2024-03-18T00:07:34.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.082+0000 7faf5be00640 1 -- 172.21.15.12:0/1385157282 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c0042a0 con 0x7faf641087e0 2024-03-18T00:07:34.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.082+0000 7faf6a600640 1 -- 172.21.15.12:0/1385157282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf641aa470 con 0x7faf641087e0 2024-03-18T00:07:34.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.083+0000 7faf6a600640 1 -- 172.21.15.12:0/1385157282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf641aa910 con 0x7faf641087e0 2024-03-18T00:07:34.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.084+0000 7faf6a600640 1 -- 172.21.15.12:0/1385157282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf6410f220 con 0x7faf641087e0 2024-03-18T00:07:34.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.087+0000 7faf5be00640 1 -- 172.21.15.12:0/1385157282 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7faf5c003970 con 0x7faf641087e0 2024-03-18T00:07:34.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.089+0000 7faf5be00640 1 --2- 172.21.15.12:0/1385157282 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7faf480a28d0 0x7faf480a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:07:34.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.089+0000 7faf5be00640 1 -- 172.21.15.12:0/1385157282 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8496+0+0 (secure 0 0 0) 0x7faf5c0f10d0 con 0x7faf641087e0 2024-03-18T00:07:34.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.090+0000 7faf62a00640 1 --2- 172.21.15.12:0/1385157282 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7faf480a28d0 0x7faf480a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:07:34.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.090+0000 7faf62a00640 1 --2- 172.21.15.12:0/1385157282 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7faf480a28d0 0x7faf480a4d90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7faf641a3ad0 tx=0x7faf5400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:07:34.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:33 smithi191 ceph-mon[36708]: pgmap v704: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-18T00:07:34.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:33 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T00:07:34.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:33 smithi191 ceph-mon[36708]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T00:07:34.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:33 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1263337311' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:34.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:33 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:34.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.096+0000 7faf5be00640 1 -- 172.21.15.12:0/1385157282 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faf5c0be030 con 0x7faf641087e0 2024-03-18T00:07:34.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:33 smithi012 ceph-mon[31541]: pgmap v704: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-18T00:07:34.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:33 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T00:07:34.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:33 smithi012 ceph-mon[31541]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T00:07:34.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:33 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1263337311' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:34.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:33 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:34.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.301+0000 7faf6a600640 1 -- 172.21.15.12:0/1385157282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7faf640623e0 con 0x7faf641087e0 2024-03-18T00:07:34.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.822+0000 7faf5be00640 1 -- 172.21.15.12:0/1385157282 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v98) ==== 178+0+0 (secure 0 0 0) 0x7faf5c0bd930 con 0x7faf641087e0 2024-03-18T00:07:34.824 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data_ec' removed 2024-03-18T00:07:34.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.828+0000 7faf6a600640 1 -- 172.21.15.12:0/1385157282 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7faf480a28d0 msgr2=0x7faf480a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:34.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.828+0000 7faf6a600640 1 --2- 172.21.15.12:0/1385157282 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7faf480a28d0 0x7faf480a4d90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7faf641a3ad0 tx=0x7faf5400b410 comp rx=0 tx=0).stop 2024-03-18T00:07:34.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.828+0000 7faf6a600640 1 -- 172.21.15.12:0/1385157282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf641087e0 msgr2=0x7faf641a25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:07:34.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.828+0000 7faf6a600640 1 --2- 172.21.15.12:0/1385157282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf641087e0 0x7faf641a25c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7faf5c005e30 tx=0x7faf5c032f10 comp rx=0 tx=0).stop 2024-03-18T00:07:34.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.828+0000 7faf6a600640 1 -- 172.21.15.12:0/1385157282 shutdown_connections 2024-03-18T00:07:34.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.828+0000 7faf6a600640 1 --2- 172.21.15.12:0/1385157282 >> [v2:172.21.15.12:6800/3179836366,v1:172.21.15.12:6801/3179836366] conn(0x7faf480a28d0 0x7faf480a4d90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:34.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.828+0000 7faf6a600640 1 --2- 172.21.15.12:0/1385157282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64110060 0x7faf641a9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:34.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.828+0000 7faf6a600640 1 --2- 172.21.15.12:0/1385157282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf64109140 0x7faf641a2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:34.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.828+0000 7faf6a600640 1 --2- 172.21.15.12:0/1385157282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf641087e0 0x7faf641a25c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:07:34.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.828+0000 7faf6a600640 1 -- 172.21.15.12:0/1385157282 >> 172.21.15.12:0/1385157282 conn(0x7faf6406e640 msgr2=0x7faf64070390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:07:34.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.828+0000 7faf6a600640 1 -- 172.21.15.12:0/1385157282 shutdown_connections 2024-03-18T00:07:34.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-18T00:07:34.828+0000 7faf6a600640 1 -- 172.21.15.12:0/1385157282 wait complete. 2024-03-18T00:07:34.857 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T00:07:34.962 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T00:07:35.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T00:07:35.046 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-18T00:07:35.046 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T00:07:35.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:34 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T00:07:35.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:34 smithi191 ceph-mon[36708]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T00:07:35.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:34 smithi191 ceph-mon[36708]: from='client.? 172.21.15.12:0/1385157282' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:35.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:34 smithi191 ceph-mon[36708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:35.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:34 smithi191 ceph-mon[36708]: pgmap v707: 33 pgs: 33 active+clean; 577 KiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:07:35.103 DEBUG:teuthology.orchestra.run.smithi079:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T00:07:35.117 DEBUG:teuthology.orchestra.run.smithi191:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T00:07:35.133 INFO:tasks.cephadm:Stopping all daemons... 2024-03-18T00:07:35.133 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-18T00:07:35.133 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.a 2024-03-18T00:07:35.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:34 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T00:07:35.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:34 smithi012 ceph-mon[31541]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T00:07:35.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:34 smithi012 ceph-mon[31541]: from='client.? 172.21.15.12:0/1385157282' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:35.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:34 smithi012 ceph-mon[31541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:35.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:34 smithi012 ceph-mon[31541]: pgmap v707: 33 pgs: 33 active+clean; 577 KiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:07:35.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:34 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T00:07:35.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:34 smithi079 ceph-mon[36241]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T00:07:35.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:34 smithi079 ceph-mon[36241]: from='client.? 172.21.15.12:0/1385157282' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:35.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:34 smithi079 ceph-mon[36241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T00:07:35.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:34 smithi079 ceph-mon[36241]: pgmap v707: 33 pgs: 33 active+clean; 577 KiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:07:35.634 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:35 smithi012 systemd[1]: Stopping Ceph mon.a for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:07:35.634 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:35 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a[31537]: 2024-03-18T00:07:35.630+0000 7f63aca00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:07:35.635 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:35 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a[31537]: 2024-03-18T00:07:35.630+0000 7f63aca00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-18T00:07:35.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:35 smithi012 podman[89453]: 2024-03-18 00:07:35.714362301 +0000 UTC m=+0.233931936 container died ca282921d1340c62ce4d0fd52fdea1dd5551421e4e7f3b2cb4d1f746bfdd9d49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-18T00:07:36.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:36 smithi012 podman[89453]: 2024-03-18 00:07:36.14164936 +0000 UTC m=+0.661219035 container cleanup ca282921d1340c62ce4d0fd52fdea1dd5551421e4e7f3b2cb4d1f746bfdd9d49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-18T00:07:36.419 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:36 smithi012 bash[89453]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a 2024-03-18T00:07:36.831 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 18 00:07:36 smithi012 podman[89467]: 2024-03-18 00:07:36.434405032 +0000 UTC m=+0.699254349 container remove ca282921d1340c62ce4d0fd52fdea1dd5551421e4e7f3b2cb4d1f746bfdd9d49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-18T00:07:36.853 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.a.service' 2024-03-18T00:07:37.043 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:07:37.043 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-18T00:07:37.043 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-18T00:07:37.043 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.b 2024-03-18T00:07:37.388 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:37 smithi079 systemd[1]: Stopping Ceph mon.b for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:07:37.795 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:37 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-b[36237]: 2024-03-18T00:07:37.387+0000 7f8a6332a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:07:37.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:37 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-b[36237]: 2024-03-18T00:07:37.387+0000 7f8a6332a700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-18T00:07:37.796 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:37 smithi079 podman[70317]: 2024-03-18 00:07:37.612983218 +0000 UTC m=+0.340851658 container died 3d8775146265547bc26009cb3e96dc7268e1e7013a112aa12c4f69591cc18137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-18T00:07:38.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:38 smithi079 podman[70317]: 2024-03-18 00:07:38.047874505 +0000 UTC m=+0.775742941 container cleanup 3d8775146265547bc26009cb3e96dc7268e1e7013a112aa12c4f69591cc18137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-b, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-18T00:07:38.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:38 smithi079 bash[70317]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-b 2024-03-18T00:07:38.572 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.b.service' 2024-03-18T00:07:38.596 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:38 smithi079 podman[70332]: 2024-03-18 00:07:38.332877538 +0000 UTC m=+0.699993265 container remove 3d8775146265547bc26009cb3e96dc7268e1e7013a112aa12c4f69591cc18137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-18T00:07:38.596 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:38 smithi079 systemd[1]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-18T00:07:38.596 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:38 smithi079 systemd[1]: Stopped Ceph mon.b for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-18T00:07:38.596 INFO:journalctl@ceph.mon.b.smithi079.stdout:Mar 18 00:07:38 smithi079 systemd[1]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.b.service: Consumed 12.385s CPU time. 2024-03-18T00:07:38.636 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:07:38.636 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-18T00:07:38.636 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-18T00:07:38.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.c 2024-03-18T00:07:38.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:38 smithi191 systemd[1]: Stopping Ceph mon.c for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:07:39.332 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:38 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-c[36702]: 2024-03-18T00:07:38.962+0000 7f74ad13e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:07:39.332 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:38 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-c[36702]: 2024-03-18T00:07:38.962+0000 7f74ad13e700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-18T00:07:39.332 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:39 smithi191 podman[71746]: 2024-03-18 00:07:39.024961061 +0000 UTC m=+0.190363611 container died 90dcfc59bc5dbd5271609613f3874d5bb096b3b9853610a001f1304eefccf4c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-18T00:07:39.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:39 smithi191 podman[71746]: 2024-03-18 00:07:39.43136703 +0000 UTC m=+0.596769565 container cleanup 90dcfc59bc5dbd5271609613f3874d5bb096b3b9853610a001f1304eefccf4c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T00:07:39.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:39 smithi191 bash[71746]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-c 2024-03-18T00:07:39.909 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.c.service' 2024-03-18T00:07:39.933 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:39 smithi191 podman[71760]: 2024-03-18 00:07:39.683144252 +0000 UTC m=+0.655933343 container remove 90dcfc59bc5dbd5271609613f3874d5bb096b3b9853610a001f1304eefccf4c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:07:39.933 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:39 smithi191 systemd[1]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-18T00:07:39.933 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:39 smithi191 systemd[1]: Stopped Ceph mon.c for 4558f3ea-e4b8-11ee-95c9-87774f69a715. 2024-03-18T00:07:39.933 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 00:07:39 smithi191 systemd[1]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mon.c.service: Consumed 12.047s CPU time. 2024-03-18T00:07:39.963 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:07:39.964 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-18T00:07:39.964 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-18T00:07:39.964 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mgr.x 2024-03-18T00:07:40.418 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 18 00:07:40 smithi012 systemd[1]: Stopping Ceph mgr.x for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:07:40.919 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 18 00:07:40 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-18T00:07:40.519+0000 7fb9ae600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:07:40.919 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 18 00:07:40 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x[31771]: 2024-03-18T00:07:40.519+0000 7fb9ae600700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-18T00:07:40.919 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 18 00:07:40 smithi012 podman[89573]: 2024-03-18 00:07:40.612199367 +0000 UTC m=+0.213799302 container died 55a544c384fb599415b45dd3ca96e825cd06af9197289d0b916c961a4cd7f307 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:07:41.389 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 18 00:07:41 smithi012 podman[89573]: 2024-03-18 00:07:41.103933058 +0000 UTC m=+0.705533010 container cleanup 55a544c384fb599415b45dd3ca96e825cd06af9197289d0b916c961a4cd7f307 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-18T00:07:41.389 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 18 00:07:41 smithi012 bash[89573]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x 2024-03-18T00:07:41.668 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Mar 18 00:07:41 smithi012 podman[89587]: 2024-03-18 00:07:41.388763989 +0000 UTC m=+0.764651369 container remove 55a544c384fb599415b45dd3ca96e825cd06af9197289d0b916c961a4cd7f307 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-18T00:07:41.950 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-18T00:07:42.083 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:07:42.083 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-18T00:07:42.084 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-18T00:07:42.084 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mgr.y 2024-03-18T00:07:42.406 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 18 00:07:42 smithi079 systemd[1]: Stopping Ceph mgr.y for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:07:42.703 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 18 00:07:42 smithi079 podman[70440]: 2024-03-18 00:07:42.406488604 +0000 UTC m=+0.129658328 container died 48dbac523852c5d1586563fe778d996b0ca9a604fd2aeea7a7e07158d28f0e0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-18T00:07:43.045 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 18 00:07:42 smithi079 podman[70440]: 2024-03-18 00:07:42.811442046 +0000 UTC m=+0.534611811 container cleanup 48dbac523852c5d1586563fe778d996b0ca9a604fd2aeea7a7e07158d28f0e0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307) 2024-03-18T00:07:43.046 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 18 00:07:42 smithi079 bash[70440]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-y 2024-03-18T00:07:43.234 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-18T00:07:43.271 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:07:43.271 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-18T00:07:43.271 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-18T00:07:43.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mgr.z 2024-03-18T00:07:43.591 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 00:07:43 smithi191 systemd[1]: Stopping Ceph mgr.z for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:07:43.896 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 00:07:43 smithi191 podman[71881]: 2024-03-18 00:07:43.614828142 +0000 UTC m=+0.159301703 container died 436c47a81ab9c856238d0210480aa4b1a899b34ec6efd03631320bacae943dfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-18T00:07:44.263 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 00:07:44 smithi191 podman[71881]: 2024-03-18 00:07:44.003217681 +0000 UTC m=+0.547691231 container cleanup 436c47a81ab9c856238d0210480aa4b1a899b34ec6efd03631320bacae943dfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-18T00:07:44.263 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 00:07:44 smithi191 bash[71881]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mgr-z 2024-03-18T00:07:44.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-18T00:07:44.467 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:07:44.468 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-18T00:07:44.468 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-18T00:07:44.468 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.0 2024-03-18T00:07:44.919 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:44 smithi012 systemd[1]: Stopping Ceph osd.0 for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:07:45.419 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:45 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0[43943]: 2024-03-18T00:07:45.076+0000 7f5fa3800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:07:45.419 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:45 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0[43943]: 2024-03-18T00:07:45.076+0000 7f5fa3800700 -1 osd.0 98 *** Got signal Terminated *** 2024-03-18T00:07:45.419 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:45 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0[43943]: 2024-03-18T00:07:45.076+0000 7f5fa3800700 -1 osd.0 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T00:07:50.989 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:50 smithi012 podman[89693]: 2024-03-18 00:07:50.680659799 +0000 UTC m=+5.777122377 container died d7b0543945540a1cb2c3c2a9fc9315e526cca5c0c97977491fc71f0332884e14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:07:51.359 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:51 smithi012 podman[89693]: 2024-03-18 00:07:51.107125805 +0000 UTC m=+6.203588391 container cleanup d7b0543945540a1cb2c3c2a9fc9315e526cca5c0c97977491fc71f0332884e14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:07:51.359 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:51 smithi012 bash[89693]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0 2024-03-18T00:07:51.668 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:51 smithi012 podman[89714]: 2024-03-18 00:07:51.358202684 +0000 UTC m=+0.667622054 container remove d7b0543945540a1cb2c3c2a9fc9315e526cca5c0c97977491fc71f0332884e14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T00:07:52.480 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:52 smithi012 podman[89803]: 2024-03-18 00:07:52.100365652 +0000 UTC m=+0.037990537 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:07:52.481 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:52 smithi012 podman[89803]: 2024-03-18 00:07:52.217435469 +0000 UTC m=+0.155060387 container create 5ad22166f6582ef4482d5770eee4e85c279c61f6af1aeedb2fbe154a95d5dc3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:07:52.481 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:52 smithi012 podman[89803]: 2024-03-18 00:07:52.479637156 +0000 UTC m=+0.417262077 container init 5ad22166f6582ef4482d5770eee4e85c279c61f6af1aeedb2fbe154a95d5dc3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:07:52.918 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:52 smithi012 podman[89803]: 2024-03-18 00:07:52.492312507 +0000 UTC m=+0.429937396 container start 5ad22166f6582ef4482d5770eee4e85c279c61f6af1aeedb2fbe154a95d5dc3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-18T00:07:52.919 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:52 smithi012 podman[89803]: 2024-03-18 00:07:52.578297292 +0000 UTC m=+0.515922287 container attach 5ad22166f6582ef4482d5770eee4e85c279c61f6af1aeedb2fbe154a95d5dc3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-18T00:07:53.335 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 18 00:07:52 smithi012 podman[89803]: 2024-03-18 00:07:52.942249972 +0000 UTC m=+0.879874894 container died 5ad22166f6582ef4482d5770eee4e85c279c61f6af1aeedb2fbe154a95d5dc3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:07:53.718 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.0.service' 2024-03-18T00:07:53.848 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:07:53.848 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-18T00:07:53.848 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-18T00:07:53.848 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.1 2024-03-18T00:07:54.157 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:07:53 smithi079 systemd[1]: Stopping Ceph osd.1 for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:07:54.545 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:07:54 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1[40845]: 2024-03-18T00:07:54.156+0000 7f11b4065700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:07:54.545 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:07:54 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1[40845]: 2024-03-18T00:07:54.156+0000 7f11b4065700 -1 osd.1 98 *** Got signal Terminated *** 2024-03-18T00:07:54.546 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:07:54 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1[40845]: 2024-03-18T00:07:54.156+0000 7f11b4065700 -1 osd.1 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T00:07:59.766 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:07:59 smithi079 podman[70576]: 2024-03-18 00:07:59.515870749 +0000 UTC m=+5.473602581 container died 4ba9a1c4a46bd7dec0ca8409a1e0eb32c03dd7114275f7237f24c4347c6904f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-18T00:08:00.045 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:07:59 smithi079 podman[70576]: 2024-03-18 00:07:59.864758228 +0000 UTC m=+5.822490051 container cleanup 4ba9a1c4a46bd7dec0ca8409a1e0eb32c03dd7114275f7237f24c4347c6904f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-18T00:08:00.046 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:07:59 smithi079 bash[70576]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1 2024-03-18T00:08:00.484 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:08:00 smithi079 podman[70593]: 2024-03-18 00:08:00.084748579 +0000 UTC m=+0.563149101 container remove 4ba9a1c4a46bd7dec0ca8409a1e0eb32c03dd7114275f7237f24c4347c6904f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-18T00:08:00.796 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:08:00 smithi079 podman[70682]: 2024-03-18 00:08:00.38716547 +0000 UTC m=+0.017412591 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:08:00.796 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:08:00 smithi079 podman[70682]: 2024-03-18 00:08:00.525477615 +0000 UTC m=+0.155724737 container create 237f1be31533f8f79b4f2e700a1f004b8a0b39f894c525ca5eec2023310a2c9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-03-18T00:08:01.216 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:08:00 smithi079 podman[70682]: 2024-03-18 00:08:00.820771927 +0000 UTC m=+0.451019055 container init 237f1be31533f8f79b4f2e700a1f004b8a0b39f894c525ca5eec2023310a2c9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-18T00:08:01.216 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:08:00 smithi079 podman[70682]: 2024-03-18 00:08:00.823825289 +0000 UTC m=+0.454072414 container start 237f1be31533f8f79b4f2e700a1f004b8a0b39f894c525ca5eec2023310a2c9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:08:01.216 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:08:00 smithi079 podman[70682]: 2024-03-18 00:08:00.928419217 +0000 UTC m=+0.558666356 container attach 237f1be31533f8f79b4f2e700a1f004b8a0b39f894c525ca5eec2023310a2c9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:08:01.498 INFO:journalctl@ceph.osd.1.smithi079.stdout:Mar 18 00:08:01 smithi079 podman[70682]: 2024-03-18 00:08:01.215968265 +0000 UTC m=+0.846215392 container died 237f1be31533f8f79b4f2e700a1f004b8a0b39f894c525ca5eec2023310a2c9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-18T00:08:01.812 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.1.service' 2024-03-18T00:08:01.853 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:08:01.853 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-18T00:08:01.853 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-18T00:08:01.853 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.2 2024-03-18T00:08:02.150 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:01 smithi191 systemd[1]: Stopping Ceph osd.2 for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:08:02.591 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:02 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2[41866]: 2024-03-18T00:08:02.148+0000 7f6c6f0a8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:08:02.591 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:02 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2[41866]: 2024-03-18T00:08:02.148+0000 7f6c6f0a8700 -1 osd.2 98 *** Got signal Terminated *** 2024-03-18T00:08:02.591 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:02 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2[41866]: 2024-03-18T00:08:02.148+0000 7f6c6f0a8700 -1 osd.2 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T00:08:07.841 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:07 smithi191 podman[72007]: 2024-03-18 00:08:07.565393047 +0000 UTC m=+5.523841382 container died e5b0aed659c11c9bbb6df55f84e6fdbe9e7d0dc25394b7a8e373d72ae8c4fa67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.schema-version=1.0) 2024-03-18T00:08:08.244 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:07 smithi191 podman[72007]: 2024-03-18 00:08:07.924877306 +0000 UTC m=+5.883325635 container cleanup e5b0aed659c11c9bbb6df55f84e6fdbe9e7d0dc25394b7a8e373d72ae8c4fa67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-18T00:08:08.244 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:07 smithi191 bash[72007]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2 2024-03-18T00:08:08.590 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:08 smithi191 podman[72025]: 2024-03-18 00:08:08.244799143 +0000 UTC m=+0.677516968 container remove e5b0aed659c11c9bbb6df55f84e6fdbe9e7d0dc25394b7a8e373d72ae8c4fa67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:08:08.841 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:08 smithi191 podman[72114]: 2024-03-18 00:08:08.493198775 +0000 UTC m=+0.017910560 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:08:08.841 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:08 smithi191 podman[72114]: 2024-03-18 00:08:08.619665365 +0000 UTC m=+0.144377150 container create 46b581a86651ef73c777ac36630302006d00538dff505f3503fbc802c51b6bb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-18T00:08:09.201 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:08 smithi191 podman[72114]: 2024-03-18 00:08:08.905831237 +0000 UTC m=+0.430543037 container init 46b581a86651ef73c777ac36630302006d00538dff505f3503fbc802c51b6bb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:08:09.202 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:08 smithi191 podman[72114]: 2024-03-18 00:08:08.909878585 +0000 UTC m=+0.434590364 container start 46b581a86651ef73c777ac36630302006d00538dff505f3503fbc802c51b6bb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:08:09.202 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:09 smithi191 podman[72114]: 2024-03-18 00:08:09.014966131 +0000 UTC m=+0.539677948 container attach 46b581a86651ef73c777ac36630302006d00538dff505f3503fbc802c51b6bb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-18T00:08:09.469 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 00:08:09 smithi191 podman[72114]: 2024-03-18 00:08:09.202026424 +0000 UTC m=+0.726738224 container died 46b581a86651ef73c777ac36630302006d00538dff505f3503fbc802c51b6bb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-18T00:08:09.853 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.2.service' 2024-03-18T00:08:09.895 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:08:09.895 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-18T00:08:09.895 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-18T00:08:09.896 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.3 2024-03-18T00:08:10.389 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:10 smithi012 systemd[1]: Stopping Ceph osd.3 for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:08:10.668 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:10 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3[50772]: 2024-03-18T00:08:10.497+0000 7fe678600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:08:10.669 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:10 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3[50772]: 2024-03-18T00:08:10.497+0000 7fe678600700 -1 osd.3 98 *** Got signal Terminated *** 2024-03-18T00:08:10.669 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:10 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3[50772]: 2024-03-18T00:08:10.497+0000 7fe678600700 -1 osd.3 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T00:08:11.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:11 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:11.180+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:12.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:12 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:12.176+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:12.841 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:12 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:12.542+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:13.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:12 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:12.854+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:13.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:13 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:13.195+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:13.806 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:13 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:13.547+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:14.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:13 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:13.806+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:14.168 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:13 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3[50772]: 2024-03-18T00:08:13.704+0000 7fe66ee00700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:44.181676+0000 front 2024-03-18T00:07:44.182003+0000 (oldest deadline 2024-03-18T00:08:13.485028+0000) 2024-03-18T00:08:14.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:14 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:14.151+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:14.668 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:14 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:14.178+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:14.786 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:14 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:14.535+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:15.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:14 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:14.786+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:15.168 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:14 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3[50772]: 2024-03-18T00:08:14.736+0000 7fe66ee00700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:44.181676+0000 front 2024-03-18T00:07:44.182003+0000 (oldest deadline 2024-03-18T00:08:13.485028+0000) 2024-03-18T00:08:15.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:15 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:15.134+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:15.668 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:15 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:15.227+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:15.841 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:15 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:15.753+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:15.841 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:15 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:15.556+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:16.257 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:15 smithi012 podman[89910]: 2024-03-18 00:08:15.970545423 +0000 UTC m=+5.613992047 container died abe7553e06dc9c0b6357fc2c4a687ede3d56bfbaaeb3626aaf304496ddf988a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:08:16.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:16 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:16.103+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:16.668 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:16 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:16.255+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:16.669 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:16 smithi012 podman[89910]: 2024-03-18 00:08:16.40602657 +0000 UTC m=+6.049473191 container cleanup abe7553e06dc9c0b6357fc2c4a687ede3d56bfbaaeb3626aaf304496ddf988a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T00:08:16.669 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:16 smithi012 bash[89910]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3 2024-03-18T00:08:16.841 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:16.573+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:16.841 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:16.731+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:17.168 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:16 smithi012 podman[89941]: 2024-03-18 00:08:16.674027218 +0000 UTC m=+0.682168023 container remove abe7553e06dc9c0b6357fc2c4a687ede3d56bfbaaeb3626aaf304496ddf988a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-18T00:08:17.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:17 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:17.126+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:17.548 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:17 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:17.208+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:17.862 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:17 smithi012 podman[90028]: 2024-03-18 00:08:17.45315052 +0000 UTC m=+0.033516421 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:08:17.862 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:17 smithi012 podman[90028]: 2024-03-18 00:08:17.591624917 +0000 UTC m=+0.171990853 container create babd5618941918e35c72bd38b70e5974e18904da53bf8af38a5baf93fcfc32f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T00:08:18.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:17 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:17.621+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:18.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:17 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:17.710+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:18.168 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:17 smithi012 podman[90028]: 2024-03-18 00:08:17.861933701 +0000 UTC m=+0.442299643 container init babd5618941918e35c72bd38b70e5974e18904da53bf8af38a5baf93fcfc32f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:08:18.169 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:17 smithi012 podman[90028]: 2024-03-18 00:08:17.873971867 +0000 UTC m=+0.454337809 container start babd5618941918e35c72bd38b70e5974e18904da53bf8af38a5baf93fcfc32f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:08:18.169 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:17 smithi012 podman[90028]: 2024-03-18 00:08:17.97738801 +0000 UTC m=+0.557753990 container attach babd5618941918e35c72bd38b70e5974e18904da53bf8af38a5baf93fcfc32f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-18T00:08:18.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:18 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:18.141+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:18.669 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 18 00:08:18 smithi012 podman[90028]: 2024-03-18 00:08:18.321376957 +0000 UTC m=+0.901742856 container died babd5618941918e35c72bd38b70e5974e18904da53bf8af38a5baf93fcfc32f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-18T00:08:18.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:18 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:18.181+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:19.051 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.3.service' 2024-03-18T00:08:19.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:18.602+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:19.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:18.743+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:19.178 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:08:19.179 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-18T00:08:19.179 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-18T00:08:19.179 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.4 2024-03-18T00:08:19.419 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:19 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:19.211+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:19.500 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:19 smithi079 systemd[1]: Stopping Ceph osd.4 for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:08:19.501 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:19 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:19.092+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:19.795 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:19 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4[47849]: 2024-03-18T00:08:19.499+0000 7f3661335700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:08:19.796 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:19 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4[47849]: 2024-03-18T00:08:19.499+0000 7f3661335700 -1 osd.4 98 *** Got signal Terminated *** 2024-03-18T00:08:19.796 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:19 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4[47849]: 2024-03-18T00:08:19.499+0000 7f3661335700 -1 osd.4 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T00:08:19.796 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:19 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4[47849]: 2024-03-18T00:08:19.704+0000 7f3659921700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.449731+0000 front 2024-03-18T00:07:58.449841+0000 (oldest deadline 2024-03-18T00:08:19.550023+0000) 2024-03-18T00:08:19.903 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:19 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:19.603+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:19.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:19 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:19.763+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:20.474 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:20 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:20.095+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:20.475 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:20 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:20.095+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:20.668 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:20 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:20.227+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:20.783 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:20.574+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:20.795 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:20 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4[47849]: 2024-03-18T00:08:20.670+0000 7f3659921700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.449731+0000 front 2024-03-18T00:07:58.449841+0000 (oldest deadline 2024-03-18T00:08:19.550023+0000) 2024-03-18T00:08:20.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:20 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:20.474+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:21.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:20.782+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:21.091 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:20.893+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:53.322826+0000 front 2024-03-18T00:07:53.322926+0000 (oldest deadline 2024-03-18T00:08:20.323453+0000) 2024-03-18T00:08:21.463 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:21 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:21.137+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:21.464 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:21 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:21.137+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:21.668 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:21 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:21.219+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:21.740 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:21 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4[47849]: 2024-03-18T00:08:21.689+0000 7f3659921700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.449731+0000 front 2024-03-18T00:07:58.449841+0000 (oldest deadline 2024-03-18T00:08:19.550023+0000) 2024-03-18T00:08:21.741 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:21 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:21.463+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:21.926 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:21 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:21.813+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:21.927 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:21 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:21.612+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:22.341 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:21 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:21.926+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:53.322826+0000 front 2024-03-18T00:07:53.322926+0000 (oldest deadline 2024-03-18T00:08:20.323453+0000) 2024-03-18T00:08:22.471 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:22 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:22.104+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:22.471 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:22 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:22.104+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:22.668 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:22 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:22.240+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:22.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:22 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:22.240+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:22.795 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:22 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4[47849]: 2024-03-18T00:08:22.705+0000 7f3659921700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.449731+0000 front 2024-03-18T00:07:58.449841+0000 (oldest deadline 2024-03-18T00:08:19.550023+0000) 2024-03-18T00:08:22.796 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:22 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:22.470+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:23.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:22 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:22.823+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:23.091 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:22 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:22.898+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:53.322826+0000 front 2024-03-18T00:07:53.322926+0000 (oldest deadline 2024-03-18T00:08:20.323453+0000) 2024-03-18T00:08:23.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:22 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:22.653+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:23.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:22 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:22.653+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:23.451 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:23 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:23.132+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:23.451 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:23 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:23.132+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:23.668 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:23 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:23.276+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:23.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:23 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:23.276+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:23.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:23 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:23.186+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:23.795 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:23 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4[47849]: 2024-03-18T00:08:23.673+0000 7f3659921700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.449731+0000 front 2024-03-18T00:07:58.449841+0000 (oldest deadline 2024-03-18T00:08:19.550023+0000) 2024-03-18T00:08:23.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:23 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:23.450+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:24.091 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:23 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:23.877+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:53.322826+0000 front 2024-03-18T00:07:53.322926+0000 (oldest deadline 2024-03-18T00:08:20.323453+0000) 2024-03-18T00:08:24.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:23 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:23.787+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:24.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:23 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:23.672+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:24.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:23 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:23.672+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:24.499 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:24 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:24.138+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:24.499 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:24 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:24.138+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:24.668 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:24 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:24.173+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:24.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:24 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:24.314+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:24.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:24 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:24.314+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:24.796 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:24 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:24.498+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:25.091 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:24 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:24.841+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:53.322826+0000 front 2024-03-18T00:07:53.322926+0000 (oldest deadline 2024-03-18T00:08:20.323453+0000) 2024-03-18T00:08:25.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:24 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:24.811+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:25.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:24 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:24.677+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:25.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:24 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:24.677+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:25.296 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:25 smithi079 podman[70787]: 2024-03-18 00:08:25.009660637 +0000 UTC m=+5.641223610 container died b2a9bcb711080d538cf06ffc2f3060f88ebd551250d1b458d14e5bc521c5321d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:08:25.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:25 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:25.155+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:25.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:25 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:25.155+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:25.418 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:25 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:25.145+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:25.419 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:25 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:25.312+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:25.419 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:25 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:25.312+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:25.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:25 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:25.471+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:25.796 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:25 smithi079 podman[70787]: 2024-03-18 00:08:25.55834439 +0000 UTC m=+6.189907354 container cleanup b2a9bcb711080d538cf06ffc2f3060f88ebd551250d1b458d14e5bc521c5321d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T00:08:25.796 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:25 smithi079 bash[70787]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4 2024-03-18T00:08:26.091 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:25 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:25.856+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:53.322826+0000 front 2024-03-18T00:07:53.322926+0000 (oldest deadline 2024-03-18T00:08:20.323453+0000) 2024-03-18T00:08:26.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:25 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:25.805+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:26.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:25 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:25.705+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:26.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:25 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:25.705+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:26.173 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:25 smithi079 podman[70816]: 2024-03-18 00:08:25.867948609 +0000 UTC m=+0.860090861 container remove b2a9bcb711080d538cf06ffc2f3060f88ebd551250d1b458d14e5bc521c5321d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T00:08:26.441 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:26 smithi079 podman[70906]: 2024-03-18 00:08:26.077224633 +0000 UTC m=+0.017493519 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:08:26.442 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:26 smithi079 podman[70906]: 2024-03-18 00:08:26.210604755 +0000 UTC m=+0.150873663 container create f6eda36ad1e513dc35d6082f027cc2983f343d8b1f92706fdef9d499de251bb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-03-18T00:08:26.442 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:26 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:26.201+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:26.442 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:26 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:26.201+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:26.668 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:26.186+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:26.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:26.354+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:26.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:26.354+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:26.755 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:26 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:26.441+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:26.756 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:26 smithi079 podman[70906]: 2024-03-18 00:08:26.471281728 +0000 UTC m=+0.411550614 container init f6eda36ad1e513dc35d6082f027cc2983f343d8b1f92706fdef9d499de251bb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-18T00:08:26.756 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:26 smithi079 podman[70906]: 2024-03-18 00:08:26.474339513 +0000 UTC m=+0.414608386 container start f6eda36ad1e513dc35d6082f027cc2983f343d8b1f92706fdef9d499de251bb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-18T00:08:26.756 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:26 smithi079 podman[70906]: 2024-03-18 00:08:26.571692681 +0000 UTC m=+0.511961563 container attach f6eda36ad1e513dc35d6082f027cc2983f343d8b1f92706fdef9d499de251bb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-18T00:08:27.045 INFO:journalctl@ceph.osd.4.smithi079.stdout:Mar 18 00:08:26 smithi079 podman[70906]: 2024-03-18 00:08:26.75661535 +0000 UTC m=+0.696884226 container died f6eda36ad1e513dc35d6082f027cc2983f343d8b1f92706fdef9d499de251bb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:08:27.091 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:26 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:26.906+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:53.322826+0000 front 2024-03-18T00:07:53.322926+0000 (oldest deadline 2024-03-18T00:08:20.323453+0000) 2024-03-18T00:08:27.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:26 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:26.817+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:27.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:26 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:26.659+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:27.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:26 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:26.659+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:27.375 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:27 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:27.184+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:27.375 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:27 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:27.184+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:27.391 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.4.service' 2024-03-18T00:08:27.432 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:08:27.433 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-18T00:08:27.433 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-18T00:08:27.433 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.5 2024-03-18T00:08:27.668 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:27 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:27.179+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:27.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:27 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:27.376+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:27.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:27 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:27.376+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:27.730 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:27 smithi191 systemd[1]: Stopping Ceph osd.5 for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:08:27.730 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:27 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:27.679+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:27.731 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:27 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:27.679+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:27.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:27 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:27.484+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:28.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:27 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:27.799+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:28.091 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:27 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:27.730+0000 7fcbf504b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:08:28.091 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:27 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:27.730+0000 7fcbf504b700 -1 osd.5 98 *** Got signal Terminated *** 2024-03-18T00:08:28.091 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:27 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:27.730+0000 7fcbf504b700 -1 osd.5 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T00:08:28.091 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:27 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:27.915+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:53.322826+0000 front 2024-03-18T00:07:53.322926+0000 (oldest deadline 2024-03-18T00:08:20.323453+0000) 2024-03-18T00:08:28.464 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:28 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:28.350+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:28.464 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:28 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:28.350+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:28.464 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:28 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:28.194+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:28.504 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:28 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:28.164+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:28.504 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:28 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:28.164+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:28.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:28 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:28.503+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:28.956 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:28 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:28.759+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:28.957 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:28 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:28.630+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:28.957 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:28 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:28.630+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:28.957 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:28 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:28.630+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:29.341 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:28 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:28.956+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:53.322826+0000 front 2024-03-18T00:07:53.322926+0000 (oldest deadline 2024-03-18T00:08:20.323453+0000) 2024-03-18T00:08:29.474 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:29 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:29.200+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:29.474 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:29 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:29.200+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:29.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:29 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:29.242+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:29.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:29 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:29.242+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:02.916336+0000 front 2024-03-18T00:08:02.916264+0000 (oldest deadline 2024-03-18T00:08:28.216527+0000) 2024-03-18T00:08:29.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:29 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:29.389+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:29.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:29 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:29.389+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:29.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:29 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:29.389+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:29.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:29 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:29.473+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:29.907 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:29 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:29.783+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:29.907 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:29 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:29.783+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:29.908 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:29 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:29.621+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:29.908 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:29 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:29.621+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:29.908 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:29 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:29.621+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:30.341 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:29 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:29.907+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:53.322826+0000 front 2024-03-18T00:07:53.322926+0000 (oldest deadline 2024-03-18T00:08:20.323453+0000) 2024-03-18T00:08:30.341 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:29 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:29.907+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.124865+0000 front 2024-03-18T00:08:06.124836+0000 (oldest deadline 2024-03-18T00:08:29.625058+0000) 2024-03-18T00:08:30.483 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:30.219+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:30.483 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:30.219+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:30.483 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:30.219+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:30.668 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:30 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:30.225+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:30.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:30 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:30.225+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:02.916336+0000 front 2024-03-18T00:08:02.916264+0000 (oldest deadline 2024-03-18T00:08:28.216527+0000) 2024-03-18T00:08:30.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:30 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:30.429+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:30.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:30 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:30.429+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:30.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:30 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:30.429+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:30.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:30 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:30.482+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:30.941 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:30.782+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:30.941 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:30.782+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:30.941 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:30.618+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:30.941 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:30.618+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:30.941 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:30.618+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:31.341 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:30.941+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:53.322826+0000 front 2024-03-18T00:07:53.322926+0000 (oldest deadline 2024-03-18T00:08:20.323453+0000) 2024-03-18T00:08:31.341 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:30 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:30.941+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.124865+0000 front 2024-03-18T00:08:06.124836+0000 (oldest deadline 2024-03-18T00:08:29.625058+0000) 2024-03-18T00:08:31.459 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:31 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:31.185+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:31.459 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:31 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:31.185+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:02.916336+0000 front 2024-03-18T00:08:02.916264+0000 (oldest deadline 2024-03-18T00:08:28.216527+0000) 2024-03-18T00:08:31.520 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:31 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:31.264+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:31.521 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:31 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:31.265+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:31.521 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:31 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:31.265+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:31.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:31 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:31.520+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:31.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:31 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:31.520+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:31.908 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:31 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:31.768+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:31.908 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:31 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:31.768+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:31.908 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:31 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:31.655+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:31.908 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:31 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:31.655+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:31.908 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:31 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:31.655+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:31.918 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:31 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:31.458+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:31.918 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:31 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:31.458+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:31.918 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:31 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:31.458+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:32.341 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:31 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:31.908+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:53.322826+0000 front 2024-03-18T00:07:53.322926+0000 (oldest deadline 2024-03-18T00:08:20.323453+0000) 2024-03-18T00:08:32.341 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:31 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5[48414]: 2024-03-18T00:08:31.908+0000 7fcbf163f700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.124865+0000 front 2024-03-18T00:08:06.124836+0000 (oldest deadline 2024-03-18T00:08:29.625058+0000) 2024-03-18T00:08:32.502 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:32 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:32.236+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:32.502 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:32 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:32.236+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:32.502 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:32 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:32.236+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:32.668 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:32 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:32.195+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:32.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:32 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:32.195+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:02.916336+0000 front 2024-03-18T00:08:02.916264+0000 (oldest deadline 2024-03-18T00:08:28.216527+0000) 2024-03-18T00:08:32.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:32 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:32.427+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:32.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:32 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:32.427+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:32.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:32 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:32.427+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:32.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:32 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:32.502+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:32.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:32 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:32.502+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:33.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:32 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:32.765+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:33.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:32 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:32.765+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:33.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:32 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:32.640+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:33.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:32 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:32.640+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:33.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:32 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:32.640+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:33.380 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:33 smithi191 podman[72231]: 2024-03-18 00:08:33.093042342 +0000 UTC m=+5.469256887 container died 4212a71e1fcda62d016a074ebac2b70400e43ece3d8d2ade9fc30d5ebf3dd30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-18T00:08:33.456 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:33 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:33.202+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:33.456 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:33 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:33.202+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:02.916336+0000 front 2024-03-18T00:08:02.916264+0000 (oldest deadline 2024-03-18T00:08:28.216527+0000) 2024-03-18T00:08:33.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:33 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:33.232+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:33.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:33 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:33.232+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:33.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:33 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:33.232+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:33.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:33 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:33.452+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:33.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:33 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:33.452+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:33.681 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:33 smithi191 podman[72231]: 2024-03-18 00:08:33.479943569 +0000 UTC m=+5.856158113 container cleanup 4212a71e1fcda62d016a074ebac2b70400e43ece3d8d2ade9fc30d5ebf3dd30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T00:08:33.681 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:33 smithi191 bash[72231]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5 2024-03-18T00:08:33.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:33 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:33.455+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:33.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:33 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:33.455+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:33.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:33 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:33.455+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:34.088 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:33 smithi191 podman[72249]: 2024-03-18 00:08:33.748091443 +0000 UTC m=+0.651329755 container remove 4212a71e1fcda62d016a074ebac2b70400e43ece3d8d2ade9fc30d5ebf3dd30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T00:08:34.088 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:33 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:33.803+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:34.088 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:33 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:33.803+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:34.089 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:33 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:33.681+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:34.089 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:33 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:33.681+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:34.089 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:33 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:33.681+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:34.341 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:34 smithi191 podman[72336]: 2024-03-18 00:08:33.99088426 +0000 UTC m=+0.017871004 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:08:34.341 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:34 smithi191 podman[72336]: 2024-03-18 00:08:34.124840457 +0000 UTC m=+0.151827193 container create e716bfab73b0859f58820ea9096d33ec8ea28c398a3ee38ee5cc9f3e0712d922 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-18T00:08:34.545 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:34 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:34.422+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:34.545 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:34 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:34.422+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:34.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:34 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:34.191+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:34.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:34 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:34.191+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:34.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:34 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:34.191+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:34.641 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:34 smithi191 podman[72336]: 2024-03-18 00:08:34.351689097 +0000 UTC m=+0.378675850 container init e716bfab73b0859f58820ea9096d33ec8ea28c398a3ee38ee5cc9f3e0712d922 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-18T00:08:34.641 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:34 smithi191 podman[72336]: 2024-03-18 00:08:34.355153025 +0000 UTC m=+0.382139760 container start e716bfab73b0859f58820ea9096d33ec8ea28c398a3ee38ee5cc9f3e0712d922 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:08:34.641 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:34 smithi191 podman[72336]: 2024-03-18 00:08:34.434880366 +0000 UTC m=+0.461867113 container attach e716bfab73b0859f58820ea9096d33ec8ea28c398a3ee38ee5cc9f3e0712d922 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5) 2024-03-18T00:08:34.668 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:34 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:34.242+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:34.668 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:34 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:34.242+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:02.916336+0000 front 2024-03-18T00:08:02.916264+0000 (oldest deadline 2024-03-18T00:08:28.216527+0000) 2024-03-18T00:08:34.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:34 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:34.464+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:34.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:34 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:34.464+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:34.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:34 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:34.464+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:35.013 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 00:08:34 smithi191 podman[72336]: 2024-03-18 00:08:34.641299574 +0000 UTC m=+0.668286313 container died e716bfab73b0859f58820ea9096d33ec8ea28c398a3ee38ee5cc9f3e0712d922 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:08:35.014 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:34 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:34.817+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:35.014 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:34 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:34.817+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:35.014 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:34 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:34.685+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:35.014 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:34 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:34.685+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:35.014 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:34 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:34.685+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:35.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.5.service' 2024-03-18T00:08:35.379 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:08:35.379 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-18T00:08:35.379 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-18T00:08:35.379 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.6 2024-03-18T00:08:35.446 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:35 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:35.177+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:35.446 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:35 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:35.177+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:35.446 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:35 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:35.177+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:35.501 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:35 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:35.235+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:35.501 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:35 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:35.235+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:02.916336+0000 front 2024-03-18T00:08:02.916264+0000 (oldest deadline 2024-03-18T00:08:28.216527+0000) 2024-03-18T00:08:35.501 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:35 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:35.423+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:35.501 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:35 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:35.423+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:35.502 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:35 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:35.423+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:35.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:35 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:35.445+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:35.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:35 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:35.445+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:35.919 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:35 smithi012 systemd[1]: Stopping Ceph osd.6 for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:08:36.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:35 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:35.694+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:36.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:35 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:35.694+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:36.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:35 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:35.694+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:36.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:35 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:35.792+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:36.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:35 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:35.792+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:36.261 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:35 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:35.996+0000 7f56ce200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:08:36.261 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:35 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:35.996+0000 7f56ce200700 -1 osd.6 98 *** Got signal Terminated *** 2024-03-18T00:08:36.262 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:35 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:35.996+0000 7f56ce200700 -1 osd.6 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T00:08:36.262 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:36 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:36.260+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:36.262 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:36 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:36.260+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:02.916336+0000 front 2024-03-18T00:08:02.916264+0000 (oldest deadline 2024-03-18T00:08:28.216527+0000) 2024-03-18T00:08:36.487 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:36 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:36.164+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:36.488 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:36 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:36.164+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:36.488 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:36 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:36.164+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:36.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:36 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:36.403+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:36.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:36 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:36.403+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:36.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:36 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:36.403+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:36.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:36 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:36.403+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:36.796 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:36 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:36.487+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:36.796 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:36 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:36.487+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:37.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:36 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:36.795+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:37.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:36 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:36.795+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:37.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:36 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:36.710+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:37.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:36 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:36.710+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:37.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:36 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:36.710+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:37.506 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:37 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:37.191+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:37.507 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:37 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:37.191+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:37.507 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:37 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:37.191+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:37.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:37 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:37.279+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:37.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:37 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:37.279+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:02.916336+0000 front 2024-03-18T00:08:02.916264+0000 (oldest deadline 2024-03-18T00:08:28.216527+0000) 2024-03-18T00:08:37.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:37 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:37.373+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:37.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:37 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:37.373+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:37.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:37 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:37.373+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:37.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:37 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:37.373+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:37.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:37 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:37.506+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:37.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:37 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:37.506+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:38.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:37 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:37.814+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:38.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:37 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:37.814+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:38.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:37 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:37.731+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:38.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:37 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:37.731+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:38.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:37 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:37.731+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:38.499 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:38 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:38.157+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:38.499 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:38 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:38.157+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:38.499 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:38 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:38.157+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:38.668 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:38 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:38.281+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:38.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:38 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:38.281+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:02.916336+0000 front 2024-03-18T00:08:02.916264+0000 (oldest deadline 2024-03-18T00:08:28.216527+0000) 2024-03-18T00:08:38.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:38 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:38.281+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.420086+0000 front 2024-03-18T00:08:13.420957+0000 (oldest deadline 2024-03-18T00:08:37.520265+0000) 2024-03-18T00:08:38.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:38 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:38.370+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:38.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:38 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:38.370+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:38.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:38 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:38.370+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:38.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:38 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:38.370+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:38.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:38 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:38.499+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:38.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:38 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:38.499+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:39.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:38 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:38.789+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:39.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:38 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:38.789+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:39.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:38 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:38.753+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:39.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:38 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:38.753+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:39.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:38 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:38.753+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:39.488 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:39 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:39.187+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:39.488 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:39 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:39.187+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:39.488 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:39 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:39.187+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:39.668 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:39 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:39.235+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:39.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:39 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:39.235+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:02.916336+0000 front 2024-03-18T00:08:02.916264+0000 (oldest deadline 2024-03-18T00:08:28.216527+0000) 2024-03-18T00:08:39.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:39 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:39.235+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.420086+0000 front 2024-03-18T00:08:13.420957+0000 (oldest deadline 2024-03-18T00:08:37.520265+0000) 2024-03-18T00:08:39.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:39 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:39.360+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:39.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:39 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:39.360+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:39.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:39 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:39.360+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:39.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:39 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:39.360+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:39.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:39 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:39.488+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:39.796 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:39 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:39.488+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:40.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:39 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:39.801+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:40.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:39 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:39.801+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:40.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:39 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:39.801+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:40.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:39 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:39.817+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:40.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:39 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:39.817+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:40.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:39 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:39.817+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:40.460 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:40 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:40.200+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:40.460 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:40 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:40.200+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:40.460 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:40 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:40.200+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:40.571 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:40 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:40.269+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.614972+0000 front 2024-03-18T00:07:57.614925+0000 (oldest deadline 2024-03-18T00:08:22.915096+0000) 2024-03-18T00:08:40.571 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:40 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:40.269+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:02.916336+0000 front 2024-03-18T00:08:02.916264+0000 (oldest deadline 2024-03-18T00:08:28.216527+0000) 2024-03-18T00:08:40.571 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:40 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6[57326]: 2024-03-18T00:08:40.269+0000 7f56c9a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.420086+0000 front 2024-03-18T00:08:13.420957+0000 (oldest deadline 2024-03-18T00:08:37.520265+0000) 2024-03-18T00:08:40.571 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:40 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:40.362+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:40.571 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:40 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:40.362+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:40.571 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:40 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:40.362+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:40.571 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:40 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:40.362+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:40.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:40 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:40.459+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:40.796 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:40 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:40.459+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:40.796 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:40 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:40.459+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:14.841578+0000 front 2024-03-18T00:08:14.841456+0000 (oldest deadline 2024-03-18T00:08:40.141313+0000) 2024-03-18T00:08:41.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:40 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:40.778+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:41.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:40 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:40.778+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:41.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:40 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:40.778+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:41.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:40 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:40.776+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:41.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:40 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:40.776+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:41.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:40 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:40.776+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:41.428 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:41 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:41.166+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:41.428 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:41 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:41.166+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:41.428 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:41 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:41.166+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:41.554 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:41 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:41.321+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:41.554 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:41 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:41.321+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:41.554 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:41 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:41.321+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:41.554 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:41 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:41.321+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:41.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:41 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:41.427+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:41.796 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:41 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:41.427+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:41.796 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:41 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:41.427+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:14.841578+0000 front 2024-03-18T00:08:14.841456+0000 (oldest deadline 2024-03-18T00:08:40.141313+0000) 2024-03-18T00:08:41.864 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:41 smithi012 podman[90131]: 2024-03-18 00:08:41.553802014 +0000 UTC m=+5.705001550 container died 101179b74bdd2df11f28e07d4ca80ddd2549ce374c500307fbbbf13098d8df6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:08:42.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:41 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:41.741+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:42.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:41 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:41.741+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:42.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:41 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:41.741+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:42.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:41 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:41.802+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:42.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:41 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:41.802+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:42.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:41 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:41.802+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:42.168 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:42 smithi012 podman[90131]: 2024-03-18 00:08:42.022575145 +0000 UTC m=+6.173774719 container cleanup 101179b74bdd2df11f28e07d4ca80ddd2549ce374c500307fbbbf13098d8df6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-03-18T00:08:42.168 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:42 smithi012 bash[90131]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6 2024-03-18T00:08:42.436 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:42 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:42.117+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:42.436 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:42 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:42.117+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:42.436 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:42 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:42.117+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:42.436 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:42 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:42.117+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:42.668 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:42 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:42.363+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:42.668 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:42 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:42.363+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:42.668 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:42 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:42.363+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:42.668 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:42 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:42.363+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:42.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:42 smithi012 podman[90161]: 2024-03-18 00:08:42.282542555 +0000 UTC m=+0.715680219 container remove 101179b74bdd2df11f28e07d4ca80ddd2549ce374c500307fbbbf13098d8df6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True) 2024-03-18T00:08:42.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:42 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:42.436+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:42.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:42 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:42.436+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:42.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:42 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:42.436+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:14.841578+0000 front 2024-03-18T00:08:14.841456+0000 (oldest deadline 2024-03-18T00:08:40.141313+0000) 2024-03-18T00:08:42.992 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:42 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:42.813+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:42.992 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:42 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:42.813+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:42.992 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:42 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:42.813+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:42.992 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:42 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:42.734+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:42.992 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:42 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:42.734+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:42.993 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:42 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:42.735+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:43.410 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:43 smithi012 podman[90252]: 2024-03-18 00:08:43.013231979 +0000 UTC m=+0.032038424 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:08:43.410 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:43 smithi012 podman[90252]: 2024-03-18 00:08:43.141573518 +0000 UTC m=+0.160380033 container create 8549bd320a5f671895b81f3baa60424a8b728a65239b1ae72de760f48bd82d87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-18T00:08:43.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:43 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:43.149+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:43.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:43 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:43.149+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:43.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:43 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:43.149+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:43.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:43 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:43.149+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:43.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:43 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:43.389+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:43.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:43 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:43.389+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:43.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:43 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:43.389+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:14.841578+0000 front 2024-03-18T00:08:14.841456+0000 (oldest deadline 2024-03-18T00:08:40.141313+0000) 2024-03-18T00:08:43.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:43 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:43.408+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:43.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:43 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:43.408+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:43.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:43 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:43.408+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:43.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:43 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:43.408+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:43.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:43 smithi012 podman[90252]: 2024-03-18 00:08:43.453964741 +0000 UTC m=+0.472771229 container init 8549bd320a5f671895b81f3baa60424a8b728a65239b1ae72de760f48bd82d87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:08:43.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:43 smithi012 podman[90252]: 2024-03-18 00:08:43.466290612 +0000 UTC m=+0.485097121 container start 8549bd320a5f671895b81f3baa60424a8b728a65239b1ae72de760f48bd82d87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-18T00:08:43.669 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:43 smithi012 podman[90252]: 2024-03-18 00:08:43.544228269 +0000 UTC m=+0.563034758 container attach 8549bd320a5f671895b81f3baa60424a8b728a65239b1ae72de760f48bd82d87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-18T00:08:44.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:43 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:43.765+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:44.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:43 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:43.765+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:44.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:43 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:43.765+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:44.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:43 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:43.769+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:44.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:43 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:43.769+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:44.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:43 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:43.769+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:44.169 INFO:journalctl@ceph.osd.6.smithi012.stdout:Mar 18 00:08:43 smithi012 podman[90252]: 2024-03-18 00:08:43.914257562 +0000 UTC m=+0.933064081 container died 8549bd320a5f671895b81f3baa60424a8b728a65239b1ae72de760f48bd82d87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-18T00:08:44.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:44 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:44.411+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:44.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:44 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:44.411+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:44.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:44 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:44.411+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:14.841578+0000 front 2024-03-18T00:08:14.841456+0000 (oldest deadline 2024-03-18T00:08:40.141313+0000) 2024-03-18T00:08:44.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:44 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:44.167+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:44.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:44 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:44.167+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:44.547 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:44 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:44.167+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:44.547 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:44 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:44.167+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:44.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:44 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:44.403+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:44.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:44 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:44.403+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:44.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:44 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:44.403+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:44.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:44 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:44.403+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:44.730 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.6.service' 2024-03-18T00:08:44.861 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:08:44.862 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-18T00:08:44.862 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-18T00:08:44.862 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.7 2024-03-18T00:08:45.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:44 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:44.795+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:45.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:44 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:44.795+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:45.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:44 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:44.795+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:45.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:44 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:44.723+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:45.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:44 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:44.723+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:45.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:44 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:44.723+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:45.178 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:44 smithi079 systemd[1]: Stopping Ceph osd.7 for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:08:45.432 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:45 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:45.192+0000 7f6ff0db3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:08:45.432 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:45 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:45.192+0000 7f6ff0db3700 -1 osd.7 98 *** Got signal Terminated *** 2024-03-18T00:08:45.432 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:45 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:45.192+0000 7f6ff0db3700 -1 osd.7 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T00:08:45.432 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:45 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:45.178+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:45.433 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:45 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:45.178+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:45.433 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:45 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:45.178+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:45.433 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:45 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:45.178+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:45.796 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:45 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:45.432+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:45.796 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:45 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:45.432+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:45.796 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:45 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:45.432+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:14.841578+0000 front 2024-03-18T00:08:14.841456+0000 (oldest deadline 2024-03-18T00:08:40.141313+0000) 2024-03-18T00:08:45.918 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:45 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:45.431+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:45.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:45 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:45.431+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:45.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:45 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:45.431+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:45.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:45 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:45.431+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:46.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:45 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:45.818+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:46.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:45 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:45.818+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:46.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:45 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:45.818+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:46.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:45 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:45.771+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:46.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:45 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:45.771+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:46.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:45 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:45.771+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:46.545 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:46 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:46.400+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:46.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:46 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:46.400+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:46.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:46 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:46.400+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:14.841578+0000 front 2024-03-18T00:08:14.841456+0000 (oldest deadline 2024-03-18T00:08:40.141313+0000) 2024-03-18T00:08:46.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:46 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:46.400+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.141810+0000 front 2024-03-18T00:08:20.141788+0000 (oldest deadline 2024-03-18T00:08:46.041946+0000) 2024-03-18T00:08:46.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:46 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:46.187+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:46.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:46 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:46.187+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:46.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:46 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:46.187+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:46.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:46 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:46.187+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:46.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:46 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:46.187+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:46.921 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:46 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:46.424+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:46.922 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:46 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:46.424+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:46.922 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:46 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:46.424+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:46.922 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:46 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:46.424+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:46.922 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:46 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:46.424+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:47.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:46 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:46.775+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:47.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:46 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:46.775+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:47.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:46 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:46.775+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:47.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:46 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:46.775+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.992603+0000 front 2024-03-18T00:08:22.992660+0000 (oldest deadline 2024-03-18T00:08:46.492722+0000) 2024-03-18T00:08:47.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:46 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:46.731+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:47.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:46 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:46.731+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:47.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:46 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:46.731+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:47.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:46 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:46.731+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:08:47.545 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:47 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:47.373+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:47.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:47 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:47.373+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:47.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:47 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:47.373+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:14.841578+0000 front 2024-03-18T00:08:14.841456+0000 (oldest deadline 2024-03-18T00:08:40.141313+0000) 2024-03-18T00:08:47.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:47 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:47.373+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.141810+0000 front 2024-03-18T00:08:20.141788+0000 (oldest deadline 2024-03-18T00:08:46.041946+0000) 2024-03-18T00:08:47.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:47 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:47.224+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:47.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:47 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:47.224+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:47.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:47 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:47.224+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:47.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:47 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:47.224+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:47.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:47 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:47.224+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:47.668 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:47.401+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:47.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:47.401+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:47.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:47.401+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:47.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:47.401+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:47.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:47 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:47.401+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:48.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:47 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:47.778+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:48.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:47 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:47.778+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:48.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:47 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:47.778+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:48.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:47 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:47.778+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.992603+0000 front 2024-03-18T00:08:22.992660+0000 (oldest deadline 2024-03-18T00:08:46.492722+0000) 2024-03-18T00:08:48.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:47 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:47.781+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:48.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:47 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:47.781+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:48.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:47 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:47.781+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:48.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:47 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:47.781+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:08:48.545 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:48 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:48.325+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:48.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:48 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:48.325+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:48.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:48 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:48.325+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:14.841578+0000 front 2024-03-18T00:08:14.841456+0000 (oldest deadline 2024-03-18T00:08:40.141313+0000) 2024-03-18T00:08:48.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:48 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:48.325+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.141810+0000 front 2024-03-18T00:08:20.141788+0000 (oldest deadline 2024-03-18T00:08:46.041946+0000) 2024-03-18T00:08:48.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:48 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:48.190+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:48.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:48 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:48.190+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:48.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:48 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:48.190+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:48.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:48 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:48.190+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:48.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:48 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:48.190+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:48.918 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:48 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:48.433+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:48.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:48 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:48.433+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:48.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:48 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:48.433+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:48.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:48 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:48.433+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:48.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:48 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:48.433+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:49.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:48 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:48.811+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:49.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:48 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:48.811+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:49.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:48 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:48.811+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:49.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:48 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:48.811+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.992603+0000 front 2024-03-18T00:08:22.992660+0000 (oldest deadline 2024-03-18T00:08:46.492722+0000) 2024-03-18T00:08:49.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:48 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:48.819+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:49.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:48 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:48.819+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:49.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:48 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:48.819+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:49.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:48 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:48.819+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:08:49.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:49 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:49.319+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:55.538710+0000 front 2024-03-18T00:07:55.538810+0000 (oldest deadline 2024-03-18T00:08:20.238894+0000) 2024-03-18T00:08:49.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:49 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:49.319+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.640164+0000 front 2024-03-18T00:08:06.640276+0000 (oldest deadline 2024-03-18T00:08:30.740204+0000) 2024-03-18T00:08:49.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:49 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:49.319+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:14.841578+0000 front 2024-03-18T00:08:14.841456+0000 (oldest deadline 2024-03-18T00:08:40.141313+0000) 2024-03-18T00:08:49.546 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:49 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7[54418]: 2024-03-18T00:08:49.319+0000 7f6fed3a7700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.141810+0000 front 2024-03-18T00:08:20.141788+0000 (oldest deadline 2024-03-18T00:08:46.041946+0000) 2024-03-18T00:08:49.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:49 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:49.171+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:49.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:49 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:49.171+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:49.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:49 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:49.171+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:49.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:49 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:49.171+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:49.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:49 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:49.171+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:49.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:49 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:49.424+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:49.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:49 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:49.424+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:49.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:49 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:49.424+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:49.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:49 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:49.424+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:49.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:49 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:49.424+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:50.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:49 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:49.837+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:50.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:49 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:49.837+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:50.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:49 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:49.837+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:50.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:49 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:49.837+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.992603+0000 front 2024-03-18T00:08:22.992660+0000 (oldest deadline 2024-03-18T00:08:46.492722+0000) 2024-03-18T00:08:50.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:49 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:49.819+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:50.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:49 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:49.819+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:50.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:49 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:49.819+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:50.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:49 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:49.819+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:08:50.529 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:50 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:50.185+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:50.529 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:50 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:50.185+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:50.529 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:50 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:50.185+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:50.529 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:50 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:50.185+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:50.529 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:50 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:50.185+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:50.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:50 smithi079 podman[71010]: 2024-03-18 00:08:50.528630789 +0000 UTC m=+5.468952958 container died 17deea3df589594098a5512c855c2c7af22ce6dff0c9489beceb785d59c4d066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-18T00:08:50.918 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:50 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:50.449+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:50.918 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:50 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:50.449+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:50.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:50 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:50.449+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:50.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:50 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:50.449+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:50.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:50 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:50.449+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:51.110 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:50 smithi079 podman[71010]: 2024-03-18 00:08:50.917919759 +0000 UTC m=+5.858241909 container cleanup 17deea3df589594098a5512c855c2c7af22ce6dff0c9489beceb785d59c4d066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:08:51.110 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:50 smithi079 bash[71010]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7 2024-03-18T00:08:51.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:50 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:50.848+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:51.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:50 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:50.848+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:51.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:50 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:50.848+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:51.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:50 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:50.848+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.992603+0000 front 2024-03-18T00:08:22.992660+0000 (oldest deadline 2024-03-18T00:08:46.492722+0000) 2024-03-18T00:08:51.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:50 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:50.846+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:51.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:50 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:50.846+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:51.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:50 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:50.846+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:51.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:50 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:50.846+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:08:51.510 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:51 smithi079 podman[71027]: 2024-03-18 00:08:51.11084445 +0000 UTC m=+0.577940274 container remove 17deea3df589594098a5512c855c2c7af22ce6dff0c9489beceb785d59c4d066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-18T00:08:51.510 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:51 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:51.137+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:51.510 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:51 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:51.137+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:51.510 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:51 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:51.137+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:51.510 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:51 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:51.137+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:51.510 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:51 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:51.137+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:51.795 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:51 smithi079 podman[71119]: 2024-03-18 00:08:51.413161637 +0000 UTC m=+0.017914998 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:08:51.796 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:51 smithi079 podman[71119]: 2024-03-18 00:08:51.547230005 +0000 UTC m=+0.151983342 container create a06c613048699afeabf3383a2056bf46a4a32fa485be9750975e1ebb35e142fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:08:51.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:51 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:51.466+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:51.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:51 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:51.466+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:51.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:51 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:51.466+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:51.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:51 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:51.466+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:51.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:51 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:51.466+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:52.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:51 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:51.848+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:52.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:51 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:51.848+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:52.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:51 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:51.848+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:52.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:51 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:51.848+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:08:52.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:51 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:51.811+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:52.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:51 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:51.811+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:52.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:51 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:51.811+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:52.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:51 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:51.811+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.992603+0000 front 2024-03-18T00:08:22.992660+0000 (oldest deadline 2024-03-18T00:08:46.492722+0000) 2024-03-18T00:08:52.102 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:51 smithi079 podman[71119]: 2024-03-18 00:08:51.814132179 +0000 UTC m=+0.418885528 container init a06c613048699afeabf3383a2056bf46a4a32fa485be9750975e1ebb35e142fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-18T00:08:52.102 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:51 smithi079 podman[71119]: 2024-03-18 00:08:51.81743435 +0000 UTC m=+0.422187690 container start a06c613048699afeabf3383a2056bf46a4a32fa485be9750975e1ebb35e142fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-03-18T00:08:52.102 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:51 smithi079 podman[71119]: 2024-03-18 00:08:51.889060529 +0000 UTC m=+0.493813887 container attach a06c613048699afeabf3383a2056bf46a4a32fa485be9750975e1ebb35e142fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-18T00:08:52.375 INFO:journalctl@ceph.osd.7.smithi079.stdout:Mar 18 00:08:52 smithi079 podman[71119]: 2024-03-18 00:08:52.102592454 +0000 UTC m=+0.707345810 container died a06c613048699afeabf3383a2056bf46a4a32fa485be9750975e1ebb35e142fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-18T00:08:52.375 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:52 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:52.121+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:52.375 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:52 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:52.121+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:52.375 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:52 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:52.121+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:52.375 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:52 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:52.121+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:52.375 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:52 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:52.121+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:52.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:52 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:52.423+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:52.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:52 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:52.423+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:52.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:52 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:52.423+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:52.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:52 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:52.423+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:52.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:52 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:52.423+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:52.722 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.7.service' 2024-03-18T00:08:52.763 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:08:52.764 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-18T00:08:52.764 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-18T00:08:52.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.8 2024-03-18T00:08:53.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:52 smithi191 systemd[1]: Stopping Ceph osd.8 for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:08:53.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:52 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:52.858+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:53.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:52 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:52.858+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:53.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:52 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:52.858+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:53.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:52 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:52.858+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.992603+0000 front 2024-03-18T00:08:22.992660+0000 (oldest deadline 2024-03-18T00:08:46.492722+0000) 2024-03-18T00:08:53.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:52 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:52.884+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:53.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:52 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:52.884+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:53.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:52 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:52.884+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:53.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:52 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:52.884+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:08:53.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:53 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:53.106+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:53.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:53 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:53.106+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:53.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:53 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:53.106+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:53.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:53 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:53.106+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:53.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:53 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:53.106+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:53.591 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:53 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:53.090+0000 7f187e44e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:08:53.591 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:53 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:53.090+0000 7f187e44e700 -1 osd.8 98 *** Got signal Terminated *** 2024-03-18T00:08:53.591 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:53 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:53.090+0000 7f187e44e700 -1 osd.8 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T00:08:53.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:53 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:53.446+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:53.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:53 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:53.446+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:53.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:53 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:53.446+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:53.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:53 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:53.446+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:53.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:53 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:53.446+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:54.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:53 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:53.884+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:54.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:53 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:53.884+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:54.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:53 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:53.884+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:54.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:53 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:53.884+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:08:54.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:53 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:53.862+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:54.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:53 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:53.862+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:54.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:53 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:53.862+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:54.092 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:53 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:53.862+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.992603+0000 front 2024-03-18T00:08:22.992660+0000 (oldest deadline 2024-03-18T00:08:46.492722+0000) 2024-03-18T00:08:54.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:54 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:54.079+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:54.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:54 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:54.079+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:54.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:54 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:54.079+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:54.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:54 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:54.079+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:54.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:54 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:54.079+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:54.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:54 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:54.441+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:54.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:54 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:54.441+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:54.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:54 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:54.441+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:54.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:54 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:54.441+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:54.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:54 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:54.441+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:55.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:54 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:54.902+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:55.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:54 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:54.902+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:55.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:54 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:54.902+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:55.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:54 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:54.902+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:08:55.342 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:54 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:54.871+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:55.342 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:54 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:54.871+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:55.342 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:54 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:54.871+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:55.342 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:54 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:54.871+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.992603+0000 front 2024-03-18T00:08:22.992660+0000 (oldest deadline 2024-03-18T00:08:46.492722+0000) 2024-03-18T00:08:55.342 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:54 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:54.871+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:29.993600+0000 front 2024-03-18T00:08:29.993626+0000 (oldest deadline 2024-03-18T00:08:54.693608+0000) 2024-03-18T00:08:55.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:55 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:55.124+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:55.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:55 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:55.124+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:55.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:55 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:55.124+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:55.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:55 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:55.124+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:55.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:55 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:55.124+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:55.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:55 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:55.465+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:55.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:55 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:55.465+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:55.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:55 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:55.465+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:55.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:55 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:55.465+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:55.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:55 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:55.465+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:55.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:55 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:55.465+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:31.354791+0000 front 2024-03-18T00:08:31.354724+0000 (oldest deadline 2024-03-18T00:08:54.854465+0000) 2024-03-18T00:08:56.342 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:55 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:55.880+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:56.342 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:55 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:55.880+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:56.342 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:55 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:55.880+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:56.342 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:55 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:55.880+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.992603+0000 front 2024-03-18T00:08:22.992660+0000 (oldest deadline 2024-03-18T00:08:46.492722+0000) 2024-03-18T00:08:56.342 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:55 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:55.880+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:29.993600+0000 front 2024-03-18T00:08:29.993626+0000 (oldest deadline 2024-03-18T00:08:54.693608+0000) 2024-03-18T00:08:56.343 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:55 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:55.948+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:56.343 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:55 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:55.948+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:56.343 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:55 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:55.948+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:56.343 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:55 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:55.948+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:08:56.343 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:55 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:55.948+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:08:56.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:56 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:56.097+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:56.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:56 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:56.097+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:56.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:56 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:56.097+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:56.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:56 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:56.097+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:56.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:56 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:56.097+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:56.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:56 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:56.480+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:56.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:56 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:56.480+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:56.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:56 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:56.480+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:56.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:56 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:56.480+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:56.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:56 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:56.480+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:56.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:56 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:56.480+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:31.354791+0000 front 2024-03-18T00:08:31.354724+0000 (oldest deadline 2024-03-18T00:08:54.854465+0000) 2024-03-18T00:08:57.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:56 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:56.918+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:57.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:56 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:56.918+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:57.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:56 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:56.918+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:57.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:56 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:56.918+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.992603+0000 front 2024-03-18T00:08:22.992660+0000 (oldest deadline 2024-03-18T00:08:46.492722+0000) 2024-03-18T00:08:57.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:56 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:56.918+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:29.993600+0000 front 2024-03-18T00:08:29.993626+0000 (oldest deadline 2024-03-18T00:08:54.693608+0000) 2024-03-18T00:08:57.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:56 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:56.961+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:57.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:56 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:56.961+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:57.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:56 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:56.961+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:57.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:56 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:56.961+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:08:57.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:56 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:56.961+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:08:57.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:57 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:57.086+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:57.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:57 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:57.086+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:57.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:57 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:57.086+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:57.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:57 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:57.086+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:57.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:57 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:57.086+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:57.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:57 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:57.086+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:08:57.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:57 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:57.449+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:57.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:57 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:57.449+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:57.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:57 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:57.449+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:57.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:57 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:57.449+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:57.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:57 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:57.449+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:57.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:57 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:57.449+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:31.354791+0000 front 2024-03-18T00:08:31.354724+0000 (oldest deadline 2024-03-18T00:08:54.854465+0000) 2024-03-18T00:08:58.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:57 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:57.897+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.387717+0000 front 2024-03-18T00:07:47.387643+0000 (oldest deadline 2024-03-18T00:08:12.087546+0000) 2024-03-18T00:08:58.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:57 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:57.897+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:57.388565+0000 front 2024-03-18T00:07:57.388587+0000 (oldest deadline 2024-03-18T00:08:22.088567+0000) 2024-03-18T00:08:58.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:57 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:57.897+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.689885+0000 front 2024-03-18T00:08:06.689943+0000 (oldest deadline 2024-03-18T00:08:28.390015+0000) 2024-03-18T00:08:58.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:57 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:57.897+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.992603+0000 front 2024-03-18T00:08:22.992660+0000 (oldest deadline 2024-03-18T00:08:46.492722+0000) 2024-03-18T00:08:58.341 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:57 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8[55295]: 2024-03-18T00:08:57.897+0000 7f187aa42700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:29.993600+0000 front 2024-03-18T00:08:29.993626+0000 (oldest deadline 2024-03-18T00:08:54.693608+0000) 2024-03-18T00:08:58.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:57 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:57.969+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:58.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:57 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:57.969+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:58.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:57 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:57.969+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:58.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:57 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:57.969+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:08:58.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:57 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:57.969+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:08:58.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:58 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:58.077+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:58.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:58 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:58.077+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:58.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:58 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:58.077+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:58.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:58 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:58.077+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:58.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:58 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:58.077+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:58.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:58 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:58.077+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:08:58.841 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:58 smithi191 podman[72453]: 2024-03-18 00:08:58.569182526 +0000 UTC m=+5.613320381 container died c0e51772638b4d0bb5f42f309b763074481831045a6ac21fc2027c7e8bd0cdb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:08:58.918 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:58 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:58.497+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:58.918 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:58 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:58.497+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:58.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:58 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:58.497+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:58.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:58 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:58.497+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:58.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:58 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:58.497+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:58.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:58 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:58.497+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:31.354791+0000 front 2024-03-18T00:08:31.354724+0000 (oldest deadline 2024-03-18T00:08:54.854465+0000) 2024-03-18T00:08:59.200 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:58 smithi191 podman[72453]: 2024-03-18 00:08:58.965928516 +0000 UTC m=+6.010066387 container cleanup c0e51772638b4d0bb5f42f309b763074481831045a6ac21fc2027c7e8bd0cdb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-18T00:08:59.200 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:58 smithi191 bash[72453]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8 2024-03-18T00:08:59.201 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:58 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:58.951+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:08:59.201 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:58 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:58.951+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:08:59.201 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:58 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:58.951+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:08:59.201 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:58 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:58.951+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:08:59.201 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:58 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:58.951+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:08:59.531 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:59 smithi191 podman[72472]: 2024-03-18 00:08:59.20043344 +0000 UTC m=+0.625342305 container remove c0e51772638b4d0bb5f42f309b763074481831045a6ac21fc2027c7e8bd0cdb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-18T00:08:59.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:59 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:59.092+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:08:59.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:59 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:59.092+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:08:59.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:59 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:59.092+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:08:59.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:59 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:59.092+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:08:59.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:59 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:59.092+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:08:59.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:08:59 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:08:59.092+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:08:59.837 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:59 smithi191 podman[72563]: 2024-03-18 00:08:59.435094878 +0000 UTC m=+0.017806171 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:08:59.837 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:59 smithi191 podman[72563]: 2024-03-18 00:08:59.551483846 +0000 UTC m=+0.134195140 container create 44bb8aa7a9ada69ed443bde482a92f20c6ae074eaafbb3820174bc076c7a432b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-18T00:08:59.837 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:59 smithi191 podman[72563]: 2024-03-18 00:08:59.837460892 +0000 UTC m=+0.420172185 container init 44bb8aa7a9ada69ed443bde482a92f20c6ae074eaafbb3820174bc076c7a432b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-18T00:08:59.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:59 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:59.466+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:08:59.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:59 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:59.466+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:08:59.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:59 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:59.466+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:08:59.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:59 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:59.466+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:08:59.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:59 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:59.466+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:08:59.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:08:59 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:08:59.466+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:31.354791+0000 front 2024-03-18T00:08:31.354724+0000 (oldest deadline 2024-03-18T00:08:54.854465+0000) 2024-03-18T00:09:00.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:59 smithi191 podman[72563]: 2024-03-18 00:08:59.84073539 +0000 UTC m=+0.423446688 container start 44bb8aa7a9ada69ed443bde482a92f20c6ae074eaafbb3820174bc076c7a432b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-18T00:09:00.091 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:08:59 smithi191 podman[72563]: 2024-03-18 00:08:59.929128138 +0000 UTC m=+0.511839448 container attach 44bb8aa7a9ada69ed443bde482a92f20c6ae074eaafbb3820174bc076c7a432b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-03-18T00:09:00.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:59 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:59.934+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:00.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:59 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:59.934+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:00.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:59 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:59.934+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:00.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:59 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:59.934+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:00.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:08:59 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:08:59.934+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:00.390 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 00:09:00 smithi191 podman[72563]: 2024-03-18 00:09:00.125630626 +0000 UTC m=+0.708341922 container died 44bb8aa7a9ada69ed443bde482a92f20c6ae074eaafbb3820174bc076c7a432b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-18T00:09:00.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:00 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:00.055+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:00.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:00 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:00.055+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:00.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:00 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:00.055+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:00.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:00 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:00.055+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:00.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:00 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:00.055+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:00.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:00 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:00.055+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:00.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.8.service' 2024-03-18T00:09:00.806 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:09:00.806 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-18T00:09:00.806 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-18T00:09:00.806 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.9 2024-03-18T00:09:00.898 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:00 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:00.497+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:09:00.898 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:00 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:00.497+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:09:00.898 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:00 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:00.497+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:09:00.898 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:00 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:00.497+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:09:00.898 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:00 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:00.497+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:09:00.898 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:00 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:00.497+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:31.354791+0000 front 2024-03-18T00:08:31.354724+0000 (oldest deadline 2024-03-18T00:08:54.854465+0000) 2024-03-18T00:09:01.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:00 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:00.963+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:01.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:00 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:00.963+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:01.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:00 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:00.963+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:01.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:00 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:00.963+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:01.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:00 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:00.963+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:01.168 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:00 smithi012 systemd[1]: Stopping Ceph osd.9 for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:09:01.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:01 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:01.031+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:01.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:01 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:01.031+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:01.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:01 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:01.031+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:01.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:01 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:01.031+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:01.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:01 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:01.031+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:01.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:01 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:01.031+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:01.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:01 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:01.394+0000 7f3bc5400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:09:01.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:01 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:01.394+0000 7f3bc5400700 -1 osd.9 98 *** Got signal Terminated *** 2024-03-18T00:09:01.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:01 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:01.394+0000 7f3bc5400700 -1 osd.9 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T00:09:01.670 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:01 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:01.468+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:09:01.670 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:01 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:01.468+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:09:01.670 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:01 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:01.468+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:09:01.670 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:01 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:01.468+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:09:01.670 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:01 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:01.468+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:09:01.670 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:01 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:01.468+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:31.354791+0000 front 2024-03-18T00:08:31.354724+0000 (oldest deadline 2024-03-18T00:08:54.854465+0000) 2024-03-18T00:09:02.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:01 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:01.998+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:02.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:01 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:01.998+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:02.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:01 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:01.998+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:02.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:01 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:01.998+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:02.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:01 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:01.998+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:02.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:01 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:01.998+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:02.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:01 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:01.983+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:02.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:01 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:01.983+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:02.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:01 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:01.983+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:02.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:01 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:01.983+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:02.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:01 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:01.983+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:02.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:01 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:01.983+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:02.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:02 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:02.474+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:09:02.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:02 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:02.474+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:09:02.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:02 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:02.474+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:09:02.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:02 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:02.474+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:09:02.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:02 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:02.474+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:09:02.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:02 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:02.474+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:31.354791+0000 front 2024-03-18T00:08:31.354724+0000 (oldest deadline 2024-03-18T00:08:54.854465+0000) 2024-03-18T00:09:03.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:02 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:02.959+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:03.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:02 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:02.959+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:03.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:02 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:02.959+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:03.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:02 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:02.959+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:03.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:02 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:02.959+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:03.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:02 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:02.959+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:03.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:02 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:02.991+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:03.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:02 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:02.991+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:03.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:02 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:02.991+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:03.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:02 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:02.991+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:03.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:02 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:02.991+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:03.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:02 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:02.991+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:03.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:03 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:03.459+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:09:03.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:03 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:03.459+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:09:03.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:03 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:03.459+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:09:03.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:03 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:03.459+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:09:03.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:03 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:03.459+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:09:03.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:03 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:03.459+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:31.354791+0000 front 2024-03-18T00:08:31.354724+0000 (oldest deadline 2024-03-18T00:08:54.854465+0000) 2024-03-18T00:09:04.295 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:03 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:03.992+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:04.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:03 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:03.992+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:04.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:03 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:03.992+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:04.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:03 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:03.992+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:04.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:03 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:03.992+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:04.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:03 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:03.992+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:04.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:03 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:03.992+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:40.860711+0000 front 2024-03-18T00:08:40.860779+0000 (oldest deadline 2024-03-18T00:09:03.760287+0000) 2024-03-18T00:09:04.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:04 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:04.005+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:04.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:04 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:04.005+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:04.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:04 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:04.005+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:04.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:04 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:04.005+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:04.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:04 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:04.005+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:04.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:04 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:04.005+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:04.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:04 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:04.486+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:09:04.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:04 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:04.486+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:09:04.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:04 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:04.486+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:09:04.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:04 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:04.486+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:09:04.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:04 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:04.486+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:09:04.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:04 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:04.486+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:31.354791+0000 front 2024-03-18T00:08:31.354724+0000 (oldest deadline 2024-03-18T00:08:54.854465+0000) 2024-03-18T00:09:05.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.022+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:05.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.022+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:05.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.022+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:05.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.022+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:05.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.022+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:05.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.022+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:05.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.022+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:40.860711+0000 front 2024-03-18T00:08:40.860779+0000 (oldest deadline 2024-03-18T00:09:03.760287+0000) 2024-03-18T00:09:05.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:05 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:05.052+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:05.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:05 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:05.052+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:05.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:05 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:05.052+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:05.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:05 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:05.052+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:05.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:05 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:05.052+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:05.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:05 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:05.052+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:05.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:05 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:05.494+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:48.239665+0000 front 2024-03-18T00:07:48.239518+0000 (oldest deadline 2024-03-18T00:08:14.139038+0000) 2024-03-18T00:09:05.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:05 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:05.494+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:58.142995+0000 front 2024-03-18T00:07:58.142828+0000 (oldest deadline 2024-03-18T00:08:21.642750+0000) 2024-03-18T00:09:05.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:05 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:05.494+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:06.945198+0000 front 2024-03-18T00:08:06.945275+0000 (oldest deadline 2024-03-18T00:08:29.245408+0000) 2024-03-18T00:09:05.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:05 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:05.494+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.848996+0000 front 2024-03-18T00:08:13.848942+0000 (oldest deadline 2024-03-18T00:08:36.149116+0000) 2024-03-18T00:09:05.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:05 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:05.494+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:20.251404+0000 front 2024-03-18T00:08:20.251696+0000 (oldest deadline 2024-03-18T00:08:45.551600+0000) 2024-03-18T00:09:05.919 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:05 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9[63931]: 2024-03-18T00:09:05.494+0000 7f3bbbc00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:31.354791+0000 front 2024-03-18T00:08:31.354724+0000 (oldest deadline 2024-03-18T00:08:54.854465+0000) 2024-03-18T00:09:06.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.991+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:06.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.991+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:06.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.991+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:06.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.991+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:06.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.991+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:06.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.991+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:06.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:05 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:05.991+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:40.860711+0000 front 2024-03-18T00:08:40.860779+0000 (oldest deadline 2024-03-18T00:09:03.760287+0000) 2024-03-18T00:09:06.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:06 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:06.059+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:06.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:06 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:06.059+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:06.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:06 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:06.059+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:06.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:06 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:06.059+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:06.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:06 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:06.059+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:06.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:06 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:06.059+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:07.168 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:06 smithi012 podman[90355]: 2024-03-18 00:09:06.895611461 +0000 UTC m=+5.646013849 container died 0c874e1ddc050a84bd7a3b4c3204d5915da8301508f25bf25012066fcf143e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-18T00:09:07.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:06 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:06.943+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:07.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:06 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:06.943+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:07.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:06 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:06.943+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:07.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:06 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:06.943+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:07.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:06 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:06.943+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:07.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:06 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:06.943+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:07.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:06 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:06.943+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:40.860711+0000 front 2024-03-18T00:08:40.860779+0000 (oldest deadline 2024-03-18T00:09:03.760287+0000) 2024-03-18T00:09:07.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:07 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:07.034+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:07.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:07 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:07.034+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:07.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:07 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:07.034+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:07.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:07 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:07.034+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:07.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:07 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:07.034+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:07.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:07 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:07.034+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:07.527 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:07 smithi012 podman[90355]: 2024-03-18 00:09:07.341989883 +0000 UTC m=+6.092392254 container cleanup 0c874e1ddc050a84bd7a3b4c3204d5915da8301508f25bf25012066fcf143e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-18T00:09:07.527 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:07 smithi012 bash[90355]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9 2024-03-18T00:09:07.918 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:07 smithi012 podman[90371]: 2024-03-18 00:09:07.526298683 +0000 UTC m=+0.620771410 container remove 0c874e1ddc050a84bd7a3b4c3204d5915da8301508f25bf25012066fcf143e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T00:09:08.295 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:07 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:07.933+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:08.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:07 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:07.933+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:08.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:07 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:07.933+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:08.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:07 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:07.933+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:08.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:07 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:07.933+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:08.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:07 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:07.933+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:08.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:07 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:07.933+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:40.860711+0000 front 2024-03-18T00:08:40.860779+0000 (oldest deadline 2024-03-18T00:09:03.760287+0000) 2024-03-18T00:09:08.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:08.021+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:08.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:08.021+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:08.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:08.021+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:08.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:08.021+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:08.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:08.021+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:08.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:08.021+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:08.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:08 smithi012 podman[90457]: 2024-03-18 00:09:08.316840865 +0000 UTC m=+0.041353024 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:09:08.669 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:08 smithi012 podman[90457]: 2024-03-18 00:09:08.427321379 +0000 UTC m=+0.151833554 container create 3c24391ca97241b9106d3a0fcc132175bb618a5c2bd80f65c41eaa65a6ce2bfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-18T00:09:09.125 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:08 smithi012 podman[90457]: 2024-03-18 00:09:08.67234351 +0000 UTC m=+0.396855691 container init 3c24391ca97241b9106d3a0fcc132175bb618a5c2bd80f65c41eaa65a6ce2bfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:09:09.125 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:08 smithi012 podman[90457]: 2024-03-18 00:09:08.682591513 +0000 UTC m=+0.407103654 container start 3c24391ca97241b9106d3a0fcc132175bb618a5c2bd80f65c41eaa65a6ce2bfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:09:09.126 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:08 smithi012 podman[90457]: 2024-03-18 00:09:08.762898097 +0000 UTC m=+0.487410279 container attach 3c24391ca97241b9106d3a0fcc132175bb618a5c2bd80f65c41eaa65a6ce2bfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:09:09.126 INFO:journalctl@ceph.osd.9.smithi012.stdout:Mar 18 00:09:09 smithi012 podman[90457]: 2024-03-18 00:09:09.125059939 +0000 UTC m=+0.849572130 container died 3c24391ca97241b9106d3a0fcc132175bb618a5c2bd80f65c41eaa65a6ce2bfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T00:09:09.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:08 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:08.901+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:09.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:08 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:08.901+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:09.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:08 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:08.901+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:09.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:08 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:08.901+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:09.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:08 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:08.901+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:09.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:08 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:08.901+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:09.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:08 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:08.901+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:40.860711+0000 front 2024-03-18T00:08:40.860779+0000 (oldest deadline 2024-03-18T00:09:03.760287+0000) 2024-03-18T00:09:09.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:08.985+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:09.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:08.985+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:09.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:08.985+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:09.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:08.985+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:09.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:08.985+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:09.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:08.985+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:09.819 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.9.service' 2024-03-18T00:09:09.934 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:09:09.934 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-18T00:09:09.934 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-18T00:09:09.934 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.10 2024-03-18T00:09:10.244 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:09 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:09.937+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:10.244 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:09 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:09.937+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:10.244 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:09 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:09.937+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:10.244 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:09 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:09.937+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:10.244 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:09 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:09.937+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:10.244 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:09 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:09.937+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:10.244 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:09 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:09.937+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:40.860711+0000 front 2024-03-18T00:08:40.860779+0000 (oldest deadline 2024-03-18T00:09:03.760287+0000) 2024-03-18T00:09:10.245 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:09 smithi079 systemd[1]: Stopping Ceph osd.10 for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:09:10.245 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:10 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:10.243+0000 7fec6eaae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:09:10.245 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:10 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:10.243+0000 7fec6eaae700 -1 osd.10 98 *** Got signal Terminated *** 2024-03-18T00:09:10.245 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:10 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:10.243+0000 7fec6eaae700 -1 osd.10 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T00:09:10.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:09 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:09.953+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:10.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:09 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:09.953+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:10.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:09 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:09.953+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:10.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:09 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:09.953+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:10.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:09 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:09.953+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:10.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:09 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:09.953+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:11.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:10 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:10.995+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:11.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:10 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:10.995+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:11.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:10 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:10.995+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:11.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:10 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:10.995+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:11.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:10 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:10.995+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:11.091 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:10 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:10.995+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:11.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:10 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:10.915+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:11.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:10 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:10.915+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:11.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:10 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:10.915+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:11.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:10 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:10.915+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:11.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:10 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:10.915+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:11.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:10 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:10.915+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:11.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:10 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:10.915+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:40.860711+0000 front 2024-03-18T00:08:40.860779+0000 (oldest deadline 2024-03-18T00:09:03.760287+0000) 2024-03-18T00:09:12.295 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:11 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:11.922+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:12.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:11 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:11.922+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:12.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:11 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:11.922+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:12.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:11 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:11.922+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:12.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:11 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:11.922+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:12.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:11 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:11.922+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:12.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:11 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:11.922+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:40.860711+0000 front 2024-03-18T00:08:40.860779+0000 (oldest deadline 2024-03-18T00:09:03.760287+0000) 2024-03-18T00:09:12.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:11 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:11.970+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:12.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:11 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:11.970+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:12.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:11 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:11.970+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:12.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:11 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:11.970+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:12.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:11 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:11.970+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:12.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:11 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:11.970+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:13.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:12 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:12.883+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:13.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:12 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:12.883+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:13.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:12 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:12.883+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:13.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:12 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:12.883+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:13.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:12 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:12.883+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:13.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:12 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:12.883+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:13.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:12 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:12.883+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:40.860711+0000 front 2024-03-18T00:08:40.860779+0000 (oldest deadline 2024-03-18T00:09:03.760287+0000) 2024-03-18T00:09:13.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:12 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:12.986+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:13.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:12 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:12.986+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:13.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:12 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:12.986+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:13.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:12 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:12.986+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:13.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:12 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:12.986+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:13.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:12 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:12.986+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:13.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:12 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:12.986+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2024-03-18T00:08:49.013314+0000 front 2024-03-18T00:08:49.013287+0000 (oldest deadline 2024-03-18T00:09:12.513324+0000) 2024-03-18T00:09:14.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:13 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:13.867+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:14.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:13 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:13.867+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:14.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:13 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:13.867+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:14.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:13 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:13.867+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:14.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:13 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:13.867+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:14.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:13 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:13.867+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:14.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:13 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:13.867+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:40.860711+0000 front 2024-03-18T00:08:40.860779+0000 (oldest deadline 2024-03-18T00:09:03.760287+0000) 2024-03-18T00:09:14.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:13 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:13.990+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:14.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:13 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:13.990+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:14.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:13 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:13.990+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:14.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:13 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:13.990+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:14.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:13 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:13.990+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:14.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:13 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:13.990+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:14.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:13 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:13.990+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2024-03-18T00:08:49.013314+0000 front 2024-03-18T00:08:49.013287+0000 (oldest deadline 2024-03-18T00:09:12.513324+0000) 2024-03-18T00:09:15.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:14 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:14.878+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:49.052549+0000 front 2024-03-18T00:07:49.052325+0000 (oldest deadline 2024-03-18T00:08:10.752166+0000) 2024-03-18T00:09:15.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:14 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:14.878+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2024-03-18T00:07:56.053116+0000 front 2024-03-18T00:07:56.053136+0000 (oldest deadline 2024-03-18T00:08:19.553217+0000) 2024-03-18T00:09:15.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:14 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:14.878+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:05.155970+0000 front 2024-03-18T00:08:05.155831+0000 (oldest deadline 2024-03-18T00:08:29.855717+0000) 2024-03-18T00:09:15.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:14 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:14.878+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.357191+0000 front 2024-03-18T00:08:13.357305+0000 (oldest deadline 2024-03-18T00:08:41.557343+0000) 2024-03-18T00:09:15.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:14 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:14.878+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:21.557724+0000 front 2024-03-18T00:08:21.557696+0000 (oldest deadline 2024-03-18T00:08:45.657720+0000) 2024-03-18T00:09:15.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:14 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:14.878+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:30.358745+0000 front 2024-03-18T00:08:30.358733+0000 (oldest deadline 2024-03-18T00:08:56.258648+0000) 2024-03-18T00:09:15.296 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:14 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10[61038]: 2024-03-18T00:09:14.878+0000 7fec6b0a2700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:40.860711+0000 front 2024-03-18T00:08:40.860779+0000 (oldest deadline 2024-03-18T00:09:03.760287+0000) 2024-03-18T00:09:15.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:14 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:14.999+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:15.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:14 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:14.999+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:15.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:14 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:14.999+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:15.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:14 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:14.999+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:15.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:14 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:14.999+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:15.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:14 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:14.999+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:15.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:14 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:14.999+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2024-03-18T00:08:49.013314+0000 front 2024-03-18T00:08:49.013287+0000 (oldest deadline 2024-03-18T00:09:12.513324+0000) 2024-03-18T00:09:16.127 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:15 smithi079 podman[71235]: 2024-03-18 00:09:15.796639545 +0000 UTC m=+5.671657190 container died b4c4727397e9d10a2d4d5b0c7f063b0134ec284327c6c534f45ffca8d45c9e66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:09:16.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:16.027+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:16.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:16.027+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:16.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:16.027+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:16.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:16.027+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:16.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:16.027+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:16.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:16.027+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:16.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:16.027+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2024-03-18T00:08:49.013314+0000 front 2024-03-18T00:08:49.013287+0000 (oldest deadline 2024-03-18T00:09:12.513324+0000) 2024-03-18T00:09:16.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:16 smithi079 podman[71235]: 2024-03-18 00:09:16.126426181 +0000 UTC m=+6.001443804 container cleanup b4c4727397e9d10a2d4d5b0c7f063b0134ec284327c6c534f45ffca8d45c9e66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-18T00:09:16.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:16 smithi079 bash[71235]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10 2024-03-18T00:09:16.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:16 smithi079 podman[71254]: 2024-03-18 00:09:16.327834688 +0000 UTC m=+0.526896037 container remove b4c4727397e9d10a2d4d5b0c7f063b0134ec284327c6c534f45ffca8d45c9e66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-18T00:09:17.046 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:16 smithi079 podman[71346]: 2024-03-18 00:09:16.638724116 +0000 UTC m=+0.017491283 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:09:17.046 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:16 smithi079 podman[71346]: 2024-03-18 00:09:16.745141161 +0000 UTC m=+0.123908326 container create ccb03ff850f64f6290d6eb8014aadeaac83c47cd186837bbb8aa3901de9434f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-18T00:09:17.046 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:16 smithi079 podman[71346]: 2024-03-18 00:09:16.980886283 +0000 UTC m=+0.359653450 container init ccb03ff850f64f6290d6eb8014aadeaac83c47cd186837bbb8aa3901de9434f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS) 2024-03-18T00:09:17.046 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:16 smithi079 podman[71346]: 2024-03-18 00:09:16.983987305 +0000 UTC m=+0.362754475 container start ccb03ff850f64f6290d6eb8014aadeaac83c47cd186837bbb8aa3901de9434f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-18T00:09:17.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:17 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:17.068+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:17.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:17 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:17.068+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:17.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:17 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:17.068+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:17.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:17 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:17.068+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:17.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:17 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:17.068+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:17.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:17 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:17.068+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:17.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:17 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:17.069+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2024-03-18T00:08:49.013314+0000 front 2024-03-18T00:08:49.013287+0000 (oldest deadline 2024-03-18T00:09:12.513324+0000) 2024-03-18T00:09:17.545 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:17 smithi079 podman[71346]: 2024-03-18 00:09:17.064114512 +0000 UTC m=+0.442881699 container attach ccb03ff850f64f6290d6eb8014aadeaac83c47cd186837bbb8aa3901de9434f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-18T00:09:17.546 INFO:journalctl@ceph.osd.10.smithi079.stdout:Mar 18 00:09:17 smithi079 podman[71346]: 2024-03-18 00:09:17.27167311 +0000 UTC m=+0.650440277 container died ccb03ff850f64f6290d6eb8014aadeaac83c47cd186837bbb8aa3901de9434f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T00:09:17.838 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.10.service' 2024-03-18T00:09:17.880 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:09:17.880 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-18T00:09:17.881 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-18T00:09:17.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.11 2024-03-18T00:09:18.202 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:17 smithi191 systemd[1]: Stopping Ceph osd.11 for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:09:18.202 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:18.075+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:18.202 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:18.075+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:18.202 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:18.076+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:18.202 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:18.076+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:18.202 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:18.076+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:18.203 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:18.076+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:18.203 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:18.076+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2024-03-18T00:08:49.013314+0000 front 2024-03-18T00:08:49.013287+0000 (oldest deadline 2024-03-18T00:09:12.513324+0000) 2024-03-18T00:09:18.203 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:18.202+0000 7fa0ce01b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:09:18.203 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:18.202+0000 7fa0ce01b700 -1 osd.11 98 *** Got signal Terminated *** 2024-03-18T00:09:18.203 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:18 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:18.202+0000 7fa0ce01b700 -1 osd.11 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T00:09:19.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:19 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:19.066+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:19.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:19 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:19.066+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:19.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:19 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:19.066+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:19.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:19 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:19.066+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:19.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:19 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:19.066+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:19.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:19 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:19.066+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:19.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:19 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:19.066+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2024-03-18T00:08:49.013314+0000 front 2024-03-18T00:08:49.013287+0000 (oldest deadline 2024-03-18T00:09:12.513324+0000) 2024-03-18T00:09:19.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:19 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:19.066+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6820 osd.8 since back 2024-03-18T00:08:53.014506+0000 front 2024-03-18T00:08:53.014494+0000 (oldest deadline 2024-03-18T00:09:18.914483+0000) 2024-03-18T00:09:20.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:20.053+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:20.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:20.053+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:20.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:20.053+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:20.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:20.053+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:20.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:20.053+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:20.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:20.053+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:20.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:20.053+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2024-03-18T00:08:49.013314+0000 front 2024-03-18T00:08:49.013287+0000 (oldest deadline 2024-03-18T00:09:12.513324+0000) 2024-03-18T00:09:20.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:20 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:20.053+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6820 osd.8 since back 2024-03-18T00:08:53.014506+0000 front 2024-03-18T00:08:53.014494+0000 (oldest deadline 2024-03-18T00:09:18.914483+0000) 2024-03-18T00:09:21.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:21 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:21.056+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:21.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:21 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:21.056+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:21.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:21 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:21.056+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:21.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:21 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:21.056+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:21.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:21 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:21.056+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:21.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:21 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:21.056+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:21.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:21 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:21.056+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2024-03-18T00:08:49.013314+0000 front 2024-03-18T00:08:49.013287+0000 (oldest deadline 2024-03-18T00:09:12.513324+0000) 2024-03-18T00:09:21.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:21 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:21.056+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6820 osd.8 since back 2024-03-18T00:08:53.014506+0000 front 2024-03-18T00:08:53.014494+0000 (oldest deadline 2024-03-18T00:09:18.914483+0000) 2024-03-18T00:09:22.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:22 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:22.083+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:22.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:22 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:22.083+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:22.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:22 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:22.083+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:22.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:22 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:22.083+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:22.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:22 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:22.083+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:22.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:22 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:22.083+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:22.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:22 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:22.083+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2024-03-18T00:08:49.013314+0000 front 2024-03-18T00:08:49.013287+0000 (oldest deadline 2024-03-18T00:09:12.513324+0000) 2024-03-18T00:09:22.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:22 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:22.083+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6820 osd.8 since back 2024-03-18T00:08:53.014506+0000 front 2024-03-18T00:08:53.014494+0000 (oldest deadline 2024-03-18T00:09:18.914483+0000) 2024-03-18T00:09:23.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:23 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:23.133+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-18T00:07:47.402171+0000 front 2024-03-18T00:07:47.402242+0000 (oldest deadline 2024-03-18T00:08:12.002930+0000) 2024-03-18T00:09:23.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:23 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:23.133+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T00:08:03.005669+0000 front 2024-03-18T00:08:03.005700+0000 (oldest deadline 2024-03-18T00:08:28.905827+0000) 2024-03-18T00:09:23.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:23 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:23.133+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-03-18T00:08:13.007148+0000 front 2024-03-18T00:08:13.007267+0000 (oldest deadline 2024-03-18T00:08:38.907037+0000) 2024-03-18T00:09:23.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:23 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:23.133+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2024-03-18T00:08:22.408077+0000 front 2024-03-18T00:08:22.408044+0000 (oldest deadline 2024-03-18T00:08:46.508147+0000) 2024-03-18T00:09:23.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:23 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:23.133+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T00:08:27.008944+0000 front 2024-03-18T00:08:27.008925+0000 (oldest deadline 2024-03-18T00:08:55.209483+0000) 2024-03-18T00:09:23.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:23 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:23.133+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-03-18T00:08:36.310942+0000 front 2024-03-18T00:08:36.311017+0000 (oldest deadline 2024-03-18T00:09:01.610796+0000) 2024-03-18T00:09:23.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:23 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:23.133+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2024-03-18T00:08:49.013314+0000 front 2024-03-18T00:08:49.013287+0000 (oldest deadline 2024-03-18T00:09:12.513324+0000) 2024-03-18T00:09:23.342 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:23 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11[62052]: 2024-03-18T00:09:23.133+0000 7fa0ca60f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.191:6820 osd.8 since back 2024-03-18T00:08:53.014506+0000 front 2024-03-18T00:08:53.014494+0000 (oldest deadline 2024-03-18T00:09:18.914483+0000) 2024-03-18T00:09:23.841 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:23 smithi191 podman[72691]: 2024-03-18 00:09:23.64317786 +0000 UTC m=+5.572208279 container died 369608f8aa6229db7e2e5ca34bd436ac65704283354f747bcb236da63af8343e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-03-18T00:09:24.276 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:24 smithi191 podman[72691]: 2024-03-18 00:09:24.000906308 +0000 UTC m=+5.929936715 container cleanup 369608f8aa6229db7e2e5ca34bd436ac65704283354f747bcb236da63af8343e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T00:09:24.277 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:24 smithi191 bash[72691]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11 2024-03-18T00:09:24.591 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:24 smithi191 podman[72710]: 2024-03-18 00:09:24.277000226 +0000 UTC m=+0.624439305 container remove 369608f8aa6229db7e2e5ca34bd436ac65704283354f747bcb236da63af8343e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T00:09:24.914 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:24 smithi191 podman[72803]: 2024-03-18 00:09:24.504189865 +0000 UTC m=+0.017558056 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:09:24.914 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:24 smithi191 podman[72803]: 2024-03-18 00:09:24.628172954 +0000 UTC m=+0.141541140 container create d33188895c7f51d2a80d702a3d44644926a5ca1e0b23e292c85699b11aa8d1fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-03-18T00:09:24.914 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:24 smithi191 podman[72803]: 2024-03-18 00:09:24.830608545 +0000 UTC m=+0.343976738 container init d33188895c7f51d2a80d702a3d44644926a5ca1e0b23e292c85699b11aa8d1fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-03-18T00:09:24.914 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:24 smithi191 podman[72803]: 2024-03-18 00:09:24.834064172 +0000 UTC m=+0.347432357 container start d33188895c7f51d2a80d702a3d44644926a5ca1e0b23e292c85699b11aa8d1fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:09:24.914 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:24 smithi191 podman[72803]: 2024-03-18 00:09:24.914109662 +0000 UTC m=+0.427477859 container attach d33188895c7f51d2a80d702a3d44644926a5ca1e0b23e292c85699b11aa8d1fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-18T00:09:25.341 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 00:09:25 smithi191 podman[72803]: 2024-03-18 00:09:25.118851605 +0000 UTC m=+0.632219800 container died d33188895c7f51d2a80d702a3d44644926a5ca1e0b23e292c85699b11aa8d1fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T00:09:25.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@osd.11.service' 2024-03-18T00:09:25.751 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:09:25.751 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-18T00:09:25.751 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-18T00:09:25.752 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.a 2024-03-18T00:09:26.168 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 18 00:09:25 smithi012 systemd[1]: Stopping Ceph mds.a for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:09:26.669 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 18 00:09:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-a[70469]: 2024-03-18T00:09:26.339+0000 7f0af5800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:09:26.669 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 18 00:09:26 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-a[70469]: 2024-03-18T00:09:26.339+0000 7f0af5800700 -1 mds.a *** got signal Terminated *** 2024-03-18T00:09:30.669 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 18 00:09:30 smithi012 podman[90573]: 2024-03-18 00:09:30.360660696 +0000 UTC m=+4.146067547 container died 1d70a05843c8fc5aa848cdbb247928db76b1b92c4d6a4630a98a725e7648e472 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:09:31.034 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 18 00:09:30 smithi012 podman[90573]: 2024-03-18 00:09:30.798889826 +0000 UTC m=+4.584296700 container cleanup 1d70a05843c8fc5aa848cdbb247928db76b1b92c4d6a4630a98a725e7648e472 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-18T00:09:31.035 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 18 00:09:30 smithi012 bash[90573]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-a 2024-03-18T00:09:31.418 INFO:journalctl@ceph.mds.a.smithi012.stdout:Mar 18 00:09:31 smithi012 podman[90587]: 2024-03-18 00:09:31.03328302 +0000 UTC m=+0.657510238 container remove 1d70a05843c8fc5aa848cdbb247928db76b1b92c4d6a4630a98a725e7648e472 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-a, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:09:31.559 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.a.service' 2024-03-18T00:09:31.686 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:09:31.686 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-18T00:09:31.687 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-18T00:09:31.687 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.d 2024-03-18T00:09:32.168 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 18 00:09:31 smithi012 systemd[1]: Stopping Ceph mds.d for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:09:32.669 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 18 00:09:32 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d[69469]: 2024-03-18T00:09:32.292+0000 7f595f600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:09:32.669 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 18 00:09:32 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d[69469]: 2024-03-18T00:09:32.292+0000 7f595f600700 -1 mds.d *** got signal Terminated *** 2024-03-18T00:09:35.216 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 18 00:09:34 smithi012 podman[90692]: 2024-03-18 00:09:34.920982765 +0000 UTC m=+2.749048731 container stop d0bed2ffb65e28296c3a377a34f5f67a0fc9cca2f710776df4f223c4ef376442 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-18T00:09:35.216 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 18 00:09:34 smithi012 podman[90692]: 2024-03-18 00:09:34.948409929 +0000 UTC m=+2.776475866 container died d0bed2ffb65e28296c3a377a34f5f67a0fc9cca2f710776df4f223c4ef376442 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-18T00:09:35.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 18 00:09:35 smithi012 podman[90692]: 2024-03-18 00:09:35.385818525 +0000 UTC m=+3.213884468 container cleanup d0bed2ffb65e28296c3a377a34f5f67a0fc9cca2f710776df4f223c4ef376442 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-18T00:09:35.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 18 00:09:35 smithi012 bash[90692]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d 2024-03-18T00:09:36.168 INFO:journalctl@ceph.mds.d.smithi012.stdout:Mar 18 00:09:35 smithi012 podman[90706]: 2024-03-18 00:09:35.703451394 +0000 UTC m=+0.760312392 container remove d0bed2ffb65e28296c3a377a34f5f67a0fc9cca2f710776df4f223c4ef376442 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-18T00:09:36.271 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.d.service' 2024-03-18T00:09:36.395 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:09:36.395 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-18T00:09:36.395 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-18T00:09:36.395 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.g 2024-03-18T00:09:36.918 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 18 00:09:36 smithi012 systemd[1]: Stopping Ceph mds.g for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:09:37.418 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 18 00:09:37 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-g[70089]: 2024-03-18T00:09:37.011+0000 7f9fc3600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:09:37.419 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 18 00:09:37 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-g[70089]: 2024-03-18T00:09:37.011+0000 7f9fc3600700 -1 mds.g *** got signal Terminated *** 2024-03-18T00:09:39.585 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 18 00:09:39 smithi012 podman[90811]: 2024-03-18 00:09:39.283681484 +0000 UTC m=+2.419351861 container died 2c8d7200ec4c560929c932af8a4eb029e7324180ad1083b1b79c90926e70aa87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-g, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-18T00:09:39.918 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 18 00:09:39 smithi012 podman[90811]: 2024-03-18 00:09:39.705349996 +0000 UTC m=+2.841020373 container cleanup 2c8d7200ec4c560929c932af8a4eb029e7324180ad1083b1b79c90926e70aa87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:09:39.919 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 18 00:09:39 smithi012 bash[90811]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-g 2024-03-18T00:09:40.418 INFO:journalctl@ceph.mds.g.smithi012.stdout:Mar 18 00:09:39 smithi012 podman[90826]: 2024-03-18 00:09:39.973247893 +0000 UTC m=+0.673424693 container remove 2c8d7200ec4c560929c932af8a4eb029e7324180ad1083b1b79c90926e70aa87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-g, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:09:40.530 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.g.service' 2024-03-18T00:09:40.649 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:09:40.649 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-18T00:09:40.649 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-18T00:09:40.649 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.j 2024-03-18T00:09:41.168 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 18 00:09:40 smithi012 systemd[1]: Stopping Ceph mds.j for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:09:41.668 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 18 00:09:41 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-j[69111]: 2024-03-18T00:09:41.206+0000 7f66aba00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:09:41.668 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 18 00:09:41 smithi012 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-j[69111]: 2024-03-18T00:09:41.206+0000 7f66aba00700 -1 mds.j *** got signal Terminated *** 2024-03-18T00:09:44.418 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 18 00:09:44 smithi012 podman[90932]: 2024-03-18 00:09:44.137817338 +0000 UTC m=+3.055149801 container died 07026a04c1a6828048de07e93ebfa8c14be36657946b74c9e5ed127e6571fea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T00:09:44.918 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 18 00:09:44 smithi012 podman[90932]: 2024-03-18 00:09:44.683577845 +0000 UTC m=+3.600910308 container cleanup 07026a04c1a6828048de07e93ebfa8c14be36657946b74c9e5ed127e6571fea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-j, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:09:44.918 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 18 00:09:44 smithi012 bash[90932]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-j 2024-03-18T00:09:45.418 INFO:journalctl@ceph.mds.j.smithi012.stdout:Mar 18 00:09:44 smithi012 podman[90946]: 2024-03-18 00:09:44.951224647 +0000 UTC m=+0.797806295 container remove 07026a04c1a6828048de07e93ebfa8c14be36657946b74c9e5ed127e6571fea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-j, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-18T00:09:45.559 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.j.service' 2024-03-18T00:09:45.670 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:09:45.671 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-18T00:09:45.671 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-18T00:09:45.671 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.b 2024-03-18T00:09:45.960 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 18 00:09:45 smithi079 systemd[1]: Stopping Ceph mds.b for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:09:46.220 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 18 00:09:45 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b[66285]: 2024-03-18T00:09:45.959+0000 7fab2b092700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:09:46.220 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 18 00:09:45 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b[66285]: 2024-03-18T00:09:45.959+0000 7fab2b092700 -1 mds.b *** got signal Terminated *** 2024-03-18T00:09:50.295 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 18 00:09:49 smithi079 podman[71451]: 2024-03-18 00:09:49.967051669 +0000 UTC m=+4.105401312 container died 90b5110dc80e26e26c77d1494b5f0767e7d7ca78eb78fecbaf2d033a257dfc85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True) 2024-03-18T00:09:50.734 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 18 00:09:50 smithi079 podman[71451]: 2024-03-18 00:09:50.425347863 +0000 UTC m=+4.563697495 container cleanup 90b5110dc80e26e26c77d1494b5f0767e7d7ca78eb78fecbaf2d033a257dfc85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T00:09:50.734 INFO:journalctl@ceph.mds.b.smithi079.stdout:Mar 18 00:09:50 smithi079 bash[71451]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-b 2024-03-18T00:09:50.853 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.b.service' 2024-03-18T00:09:50.893 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:09:50.893 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-18T00:09:50.893 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-18T00:09:50.893 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.e 2024-03-18T00:09:51.045 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 18 00:09:50 smithi079 systemd[1]: Stopping Ceph mds.e for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:09:51.545 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 18 00:09:51 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-e[65654]: 2024-03-18T00:09:51.245+0000 7f7158a6d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:09:51.545 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 18 00:09:51 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-e[65654]: 2024-03-18T00:09:51.245+0000 7f7158a6d700 -1 mds.e *** got signal Terminated *** 2024-03-18T00:09:55.289 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 18 00:09:54 smithi079 podman[71582]: 2024-03-18 00:09:54.865332492 +0000 UTC m=+3.761419526 container died c3b0811d0d33bceb4607f580ba60889522d087bbe244500c3f0b48cb39a73f2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-e, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:09:55.545 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 18 00:09:55 smithi079 podman[71582]: 2024-03-18 00:09:55.372491176 +0000 UTC m=+4.268578253 container cleanup c3b0811d0d33bceb4607f580ba60889522d087bbe244500c3f0b48cb39a73f2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-e, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T00:09:55.546 INFO:journalctl@ceph.mds.e.smithi079.stdout:Mar 18 00:09:55 smithi079 bash[71582]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-e 2024-03-18T00:09:55.813 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.e.service' 2024-03-18T00:09:55.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:09:55.849 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-18T00:09:55.849 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-18T00:09:55.849 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.h 2024-03-18T00:09:56.167 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 18 00:09:55 smithi079 systemd[1]: Stopping Ceph mds.h for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:09:56.545 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 18 00:09:56 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-h[65960]: 2024-03-18T00:09:56.167+0000 7fdda9de7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:09:56.545 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 18 00:09:56 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-h[65960]: 2024-03-18T00:09:56.167+0000 7fdda9de7700 -1 mds.h *** got signal Terminated *** 2024-03-18T00:09:59.235 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 18 00:09:58 smithi079 podman[71701]: 2024-03-18 00:09:58.871107795 +0000 UTC m=+2.811028637 container died 7386f8afc90f03e7705193fd0d4da241f6427bf348b792ce8fecedd0107a92ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-18T00:09:59.545 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 18 00:09:59 smithi079 podman[71701]: 2024-03-18 00:09:59.343056403 +0000 UTC m=+3.282977177 container cleanup 7386f8afc90f03e7705193fd0d4da241f6427bf348b792ce8fecedd0107a92ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-h, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:09:59.545 INFO:journalctl@ceph.mds.h.smithi079.stdout:Mar 18 00:09:59 smithi079 bash[71701]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-h 2024-03-18T00:09:59.709 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.h.service' 2024-03-18T00:09:59.745 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:09:59.745 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-18T00:09:59.745 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-18T00:09:59.746 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.k 2024-03-18T00:10:00.045 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 18 00:09:59 smithi079 systemd[1]: Stopping Ceph mds.k for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:10:00.546 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 18 00:10:00 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-k[65337]: 2024-03-18T00:10:00.071+0000 7f8b71c57700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:10:00.546 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 18 00:10:00 smithi079 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-k[65337]: 2024-03-18T00:10:00.071+0000 7f8b71c57700 -1 mds.k *** got signal Terminated *** 2024-03-18T00:10:03.132 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 18 00:10:02 smithi079 podman[71823]: 2024-03-18 00:10:02.872689004 +0000 UTC m=+2.915467639 container died 08729b28bcffeea94e32916fd3df8a67ce5e69c3dc0678055a6b4f6e9cfef551 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-k, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-18T00:10:03.491 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 18 00:10:03 smithi079 podman[71823]: 2024-03-18 00:10:03.231805677 +0000 UTC m=+3.274584319 container cleanup 08729b28bcffeea94e32916fd3df8a67ce5e69c3dc0678055a6b4f6e9cfef551 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-18T00:10:03.491 INFO:journalctl@ceph.mds.k.smithi079.stdout:Mar 18 00:10:03 smithi079 bash[71823]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-k 2024-03-18T00:10:03.652 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.k.service' 2024-03-18T00:10:03.687 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:10:03.687 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-18T00:10:03.688 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-18T00:10:03.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.c 2024-03-18T00:10:03.990 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 00:10:03 smithi191 systemd[1]: Stopping Ceph mds.c for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:10:04.341 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 00:10:03 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-c[67504]: 2024-03-18T00:10:03.988+0000 7fb17a24b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:10:04.341 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 00:10:03 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-c[67504]: 2024-03-18T00:10:03.988+0000 7fb17a24b700 -1 mds.c *** got signal Terminated *** 2024-03-18T00:10:07.323 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 00:10:06 smithi191 podman[72910]: 2024-03-18 00:10:06.867591383 +0000 UTC m=+2.984428929 container died 949e250428f268f3cef42d445aa6b225b467c426bc3cb4d090d2e8b11544e6d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-c, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-18T00:10:07.591 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 00:10:07 smithi191 podman[72910]: 2024-03-18 00:10:07.422385315 +0000 UTC m=+3.539222854 container cleanup 949e250428f268f3cef42d445aa6b225b467c426bc3cb4d090d2e8b11544e6d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS) 2024-03-18T00:10:07.591 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 00:10:07 smithi191 bash[72910]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-c 2024-03-18T00:10:07.821 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.c.service' 2024-03-18T00:10:07.861 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:10:07.861 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-18T00:10:07.861 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-18T00:10:07.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.f 2024-03-18T00:10:08.091 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 00:10:07 smithi191 systemd[1]: Stopping Ceph mds.f for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:10:08.591 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 00:10:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-f[67197]: 2024-03-18T00:10:08.183+0000 7f2c56ba1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:10:08.591 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 00:10:08 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-f[67197]: 2024-03-18T00:10:08.183+0000 7f2c56ba1700 -1 mds.f *** got signal Terminated *** 2024-03-18T00:10:11.135 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 00:10:10 smithi191 podman[73044]: 2024-03-18 00:10:10.86256581 +0000 UTC m=+2.784182009 container died fad973357eed9716973bcce61deb5bf2e171ffa2c85c2fdccdc99e031d31ba68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-f, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-18T00:10:11.477 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 00:10:11 smithi191 podman[73044]: 2024-03-18 00:10:11.226046496 +0000 UTC m=+3.147662698 container cleanup fad973357eed9716973bcce61deb5bf2e171ffa2c85c2fdccdc99e031d31ba68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-f, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-18T00:10:11.477 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 00:10:11 smithi191 bash[73044]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-f 2024-03-18T00:10:11.646 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.f.service' 2024-03-18T00:10:11.682 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:10:11.682 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-18T00:10:11.682 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-18T00:10:11.682 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.i 2024-03-18T00:10:11.996 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 00:10:11 smithi191 systemd[1]: Stopping Ceph mds.i for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:10:12.341 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 00:10:11 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-i[67817]: 2024-03-18T00:10:11.996+0000 7f0224159700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:10:12.341 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 00:10:11 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-i[67817]: 2024-03-18T00:10:11.996+0000 7f0224159700 -1 mds.i *** got signal Terminated *** 2024-03-18T00:10:15.091 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 00:10:14 smithi191 podman[73165]: 2024-03-18 00:10:14.85647397 +0000 UTC m=+2.961926222 container died 8439bf52950881deea7596b5c981f7eb814d2713d9c46d99e4b4f7c5903156ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-i, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:10:15.488 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 00:10:15 smithi191 podman[73165]: 2024-03-18 00:10:15.221131953 +0000 UTC m=+3.326584198 container cleanup 8439bf52950881deea7596b5c981f7eb814d2713d9c46d99e4b4f7c5903156ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-i, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T00:10:15.488 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 00:10:15 smithi191 bash[73165]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-i 2024-03-18T00:10:15.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.i.service' 2024-03-18T00:10:15.686 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:10:15.686 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-18T00:10:15.686 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-18T00:10:15.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.l 2024-03-18T00:10:16.008 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 00:10:15 smithi191 systemd[1]: Stopping Ceph mds.l for 4558f3ea-e4b8-11ee-95c9-87774f69a715... 2024-03-18T00:10:16.341 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 00:10:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-l[66890]: 2024-03-18T00:10:16.008+0000 7fa4f0b52700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:10:16.342 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 00:10:16 smithi191 ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-l[66890]: 2024-03-18T00:10:16.008+0000 7fa4f0b52700 -1 mds.l *** got signal Terminated *** 2024-03-18T00:10:19.141 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 00:10:18 smithi191 podman[73287]: 2024-03-18 00:10:18.866480973 +0000 UTC m=+2.954829569 container died b100e76a6ff3149cfd49771a3f27aa79be85182835834dba13be31678ab8cccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:10:19.400 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 00:10:19 smithi191 podman[73287]: 2024-03-18 00:10:19.207922931 +0000 UTC m=+3.296271529 container cleanup b100e76a6ff3149cfd49771a3f27aa79be85182835834dba13be31678ab8cccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-l, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-18T00:10:19.400 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 00:10:19 smithi191 bash[73287]: ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715-mds-l 2024-03-18T00:10:19.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4558f3ea-e4b8-11ee-95c9-87774f69a715@mds.l.service' 2024-03-18T00:10:19.655 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T00:10:19.655 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-18T00:10:19.655 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T00:10:19.968 INFO:teuthology.orchestra.run.smithi012.stdout:Deleting cluster with fsid: 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-18T00:10:25.034 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T00:10:25.225 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-18T00:10:27.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T00:10:28.006 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-18T00:10:30.485 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T00:10:30.583 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T00:10:30.618 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T00:10:30.647 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-18T00:10:30.649 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463/remote/smithi012/crash 2024-03-18T00:10:30.650 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/crash -- . 2024-03-18T00:10:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:tar: /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T00:10:30.747 INFO:teuthology.orchestra.run.smithi012.stderr:tar: Error is not recoverable: exiting now 2024-03-18T00:10:30.752 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463/remote/smithi079/crash 2024-03-18T00:10:30.753 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/crash -- . 2024-03-18T00:10:30.794 INFO:teuthology.orchestra.run.smithi079.stderr:tar: /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T00:10:30.795 INFO:teuthology.orchestra.run.smithi079.stderr:tar: Error is not recoverable: exiting now 2024-03-18T00:10:30.797 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463/remote/smithi191/crash 2024-03-18T00:10:30.798 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/crash -- . 2024-03-18T00:10:30.848 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T00:10:30.849 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now 2024-03-18T00:10:30.849 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-18T00:10:30.850 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-18T00:10:30.961 INFO:teuthology.orchestra.run.smithi012.stdout:2024-03-17T23:48:03.070702+0000 mon.a (mon.0) 210 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] is down (out of quorum) 2024-03-18T00:10:30.963 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-18T00:10:30.963 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[SEC\]' /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-18T00:10:31.053 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]' /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-18T00:10:31.143 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[WRN\]' /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-18T00:10:31.228 INFO:teuthology.orchestra.run.smithi012.stdout:2024-03-17T23:48:03.070702+0000 mon.a (mon.0) 210 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] is down (out of quorum) 2024-03-18T00:10:31.229 INFO:tasks.cephadm:Compressing logs... 2024-03-18T00:10:31.230 DEBUG:teuthology.orchestra.run.smithi012:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T00:10:31.234 DEBUG:teuthology.orchestra.run.smithi079:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T00:10:31.237 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T00:10:31.265 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T00:10:31.266 INFO:teuthology.orchestra.run.smithi191.stderr:: No such file or directory 2024-03-18T00:10:31.266 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-18T00:10:31.266 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T00:10:31.266 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-18T00:10:31.266 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-18T00:10:31.266 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-18T00:10:31.271 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.i.log: /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-18T00:10:31.271 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T00:10:31.271 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-18T00:10:31.271 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-18T00:10:31.271 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T00:10:31.271 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.f.log: 90.3% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T00:10:31.271 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.log 2024-03-18T00:10:31.271 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.cephadm.log: 82.0% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T00:10:31.271 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-18T00:10:31.274 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.log: 86.6% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T00:10:31.274 INFO:teuthology.orchestra.run.smithi191.stderr: 92.8% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mgr.z.log.gzgzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-18T00:10:31.274 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-18T00:10:31.274 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T00:10:31.284 INFO:teuthology.orchestra.run.smithi079.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T00:10:31.284 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T00:10:31.284 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-18T00:10:31.288 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T00:10:31.288 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-18T00:10:31.288 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-18T00:10:31.288 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.audit.log: /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-18T00:10:31.288 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-18T00:10:31.289 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-18T00:10:31.289 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-18T00:10:31.289 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-18T00:10:31.289 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-18T00:10:31.289 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T00:10:31.289 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.e.log: /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.log 2024-03-18T00:10:31.289 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T00:10:31.289 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-18T00:10:31.290 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-18T00:10:31.292 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-18T00:10:31.302 INFO:teuthology.orchestra.run.smithi012.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T00:10:31.302 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T00:10:31.303 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-18T00:10:31.304 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-18T00:10:31.305 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T00:10:31.306 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-18T00:10:31.308 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-18T00:10:31.310 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-18T00:10:31.312 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-18T00:10:31.312 INFO:teuthology.orchestra.run.smithi012.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T00:10:31.312 INFO:teuthology.orchestra.run.smithi012.stderr: 90.1% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T00:10:31.313 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-18T00:10:31.314 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T00:10:31.315 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T00:10:31.316 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.0.log: 94.5% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-18T00:10:31.317 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.cephadm.log: 94.6% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-18T00:10:31.317 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.log 2024-03-18T00:10:31.317 INFO:teuthology.orchestra.run.smithi012.stderr: 82.4% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T00:10:31.318 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T00:10:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-18T00:10:31.323 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-18T00:10:31.323 INFO:teuthology.orchestra.run.smithi012.stderr: 86.6% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T00:10:31.324 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-18T00:10:31.327 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.d.log: /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-18T00:10:31.337 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-18T00:10:31.338 INFO:teuthology.orchestra.run.smithi012.stderr: 90.8% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T00:10:31.377 INFO:teuthology.orchestra.run.smithi079.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T00:10:31.425 INFO:teuthology.orchestra.run.smithi079.stderr: 82.0% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T00:10:31.491 INFO:teuthology.orchestra.run.smithi079.stderr: 90.3% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T00:10:31.493 INFO:teuthology.orchestra.run.smithi191.stderr: 94.8% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-18T00:10:31.584 INFO:teuthology.orchestra.run.smithi079.stderr: 86.6% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T00:10:31.584 INFO:teuthology.orchestra.run.smithi079.stderr: 90.9% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T00:10:31.588 INFO:teuthology.orchestra.run.smithi012.stderr: 90.1% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-18T00:10:31.813 INFO:teuthology.orchestra.run.smithi191.stderr: 95.0% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-18T00:10:31.838 INFO:teuthology.orchestra.run.smithi191.stderr: 91.9% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-18T00:10:32.547 INFO:teuthology.orchestra.run.smithi079.stderr: 94.7% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-18T00:10:32.554 INFO:teuthology.orchestra.run.smithi012.stderr: 90.3% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-18T00:10:33.215 INFO:teuthology.orchestra.run.smithi079.stderr: 91.8% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-18T00:10:34.763 INFO:teuthology.orchestra.run.smithi191.stderr: 94.2% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-18T00:10:34.841 INFO:teuthology.orchestra.run.smithi079.stderr: 95.1% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-18T00:10:37.610 INFO:teuthology.orchestra.run.smithi191.stderr: 94.2% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-18T00:10:40.887 INFO:teuthology.orchestra.run.smithi012.stderr: 94.1% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-18T00:10:47.038 INFO:teuthology.orchestra.run.smithi191.stderr: 91.8% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-18T00:10:47.520 INFO:teuthology.orchestra.run.smithi012.stderr: 94.8% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-18T00:10:51.115 INFO:teuthology.orchestra.run.smithi191.stderr: 91.4% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-18T00:10:52.545 INFO:teuthology.orchestra.run.smithi191.stderr: 91.3% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-18T00:10:53.447 INFO:teuthology.orchestra.run.smithi012.stderr: 91.5% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-18T00:10:54.488 INFO:teuthology.orchestra.run.smithi012.stderr: 91.4% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-18T00:10:55.114 INFO:teuthology.orchestra.run.smithi079.stderr: 94.4% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-18T00:10:58.376 INFO:teuthology.orchestra.run.smithi012.stderr: 91.3% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-18T00:11:00.060 INFO:teuthology.orchestra.run.smithi191.stderr: 91.3% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-18T00:11:00.062 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-18T00:11:00.062 INFO:teuthology.orchestra.run.smithi191.stderr:real 0m28.810s 2024-03-18T00:11:00.062 INFO:teuthology.orchestra.run.smithi191.stderr:user 1m33.616s 2024-03-18T00:11:00.062 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m2.874s 2024-03-18T00:11:01.700 INFO:teuthology.orchestra.run.smithi012.stderr: 91.5% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-18T00:11:01.706 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-03-18T00:11:01.706 INFO:teuthology.orchestra.run.smithi012.stderr:real 0m30.439s 2024-03-18T00:11:01.706 INFO:teuthology.orchestra.run.smithi012.stderr:user 2m1.651s 2024-03-18T00:11:01.706 INFO:teuthology.orchestra.run.smithi012.stderr:sys 0m8.473s 2024-03-18T00:11:08.978 INFO:teuthology.orchestra.run.smithi079.stderr: 91.8% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-18T00:11:14.634 INFO:teuthology.orchestra.run.smithi079.stderr: 91.6% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-18T00:11:17.496 INFO:teuthology.orchestra.run.smithi079.stderr: 91.5% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-18T00:11:46.829 INFO:teuthology.orchestra.run.smithi079.stderr: 91.5% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-18T00:14:53.653 INFO:teuthology.orchestra.run.smithi079.stderr: 93.0% -- replaced with /var/log/ceph/4558f3ea-e4b8-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-18T00:14:53.654 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-18T00:14:53.655 INFO:teuthology.orchestra.run.smithi079.stderr:real 4m22.403s 2024-03-18T00:14:53.655 INFO:teuthology.orchestra.run.smithi079.stderr:user 4m40.996s 2024-03-18T00:14:53.655 INFO:teuthology.orchestra.run.smithi079.stderr:sys 0m9.945s 2024-03-18T00:14:53.655 INFO:tasks.cephadm:Archiving logs... 2024-03-18T00:14:53.656 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463/remote/smithi012/log 2024-03-18T00:14:53.657 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T00:15:11.255 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463/remote/smithi079/log 2024-03-18T00:15:11.256 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T00:16:00.222 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463/remote/smithi191/log 2024-03-18T00:16:00.223 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T00:16:15.659 INFO:tasks.cephadm:Removing cluster... 2024-03-18T00:16:15.659 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 --force 2024-03-18T00:16:15.998 INFO:teuthology.orchestra.run.smithi012.stdout:Deleting cluster with fsid: 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-18T00:16:17.039 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 --force 2024-03-18T00:16:17.233 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-18T00:16:17.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4558f3ea-e4b8-11ee-95c9-87774f69a715 --force 2024-03-18T00:16:17.983 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: 4558f3ea-e4b8-11ee-95c9-87774f69a715 2024-03-18T00:16:18.422 INFO:tasks.cephadm:Removing cephadm ... 2024-03-18T00:16:18.422 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T00:16:18.483 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T00:16:18.501 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T00:16:18.517 INFO:tasks.cephadm:Teardown complete 2024-03-18T00:16:18.517 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-18T00:16:18.549 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-18T00:16:18.723 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T00:16:18.734 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T00:16:18.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T00:16:18.908 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-18T00:16:18.978 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-18T00:16:18.978 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T00:16:18.988 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T00:16:18.992 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T00:16:19.006 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-03-18T00:16:19.006 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-18T00:16:19.011 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T00:16:19.011 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-18T00:16:19.011 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 +1226us[+1428us] +/- 89ms 2024-03-18T00:16:19.011 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -250us[ -47us] +/- 74ms 2024-03-18T00:16:19.011 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -587us[ -384us] +/- 65ms 2024-03-18T00:16:19.011 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:16:19.039 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-03-18T00:16:19.049 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T00:16:19.049 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-03-18T00:16:19.049 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 +1409us[+1409us] +/- 89ms 2024-03-18T00:16:19.050 INFO:teuthology.orchestra.run.smithi012.stdout:^* hv02.front.sepia.ceph.com 3 6 377 63 -77us[ -96us] +/- 74ms 2024-03-18T00:16:19.050 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -415us[ -415us] +/- 65ms 2024-03-18T00:16:19.050 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:16:19.056 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T00:16:19.056 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-03-18T00:16:19.056 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 +1392us[+1392us] +/- 89ms 2024-03-18T00:16:19.056 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -101us[ -143us] +/- 74ms 2024-03-18T00:16:19.057 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 -434us[ -434us] +/- 65ms 2024-03-18T00:16:19.057 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:16:19.066 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-18T00:16:19.075 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-18T00:16:19.076 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-18T00:16:19.086 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T00:16:19.337 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T00:16:19.432 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T00:16:19.507 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T00:16:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:16:19.618 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T00:16:19.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:16:19.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T00:16:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:16:19.685 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-18T00:16:19.696 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-18T00:16:19.706 INFO:teuthology.task.internal:Duration was 2829.196415 seconds 2024-03-18T00:16:19.706 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-18T00:16:19.718 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-18T00:16:19.718 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T00:16:19.726 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T00:16:19.729 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T00:16:19.772 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T00:16:19.793 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T00:16:19.899 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T00:16:20.255 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-18T00:16:20.256 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2024-03-18T00:16:20.256 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T00:16:20.318 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-03-18T00:16:20.318 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T00:16:20.351 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-03-18T00:16:20.351 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T00:16:20.370 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-18T00:16:20.370 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T00:16:20.378 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T00:16:20.394 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T00:16:20.474 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-18T00:16:20.474 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T00:16:20.479 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T00:16:20.481 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T00:16:21.007 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-18T00:16:21.018 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-18T00:16:21.018 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T00:16:21.111 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T00:16:21.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T00:16:21.181 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-18T00:16:21.192 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T00:16:21.200 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T00:16:21.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T00:16:21.230 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-03-18T00:16:21.247 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-03-18T00:16:21.308 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2024-03-18T00:16:21.363 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T00:16:21.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:16:21.415 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T00:16:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:16:21.429 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T00:16:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:16:21.444 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-18T00:16:21.456 INFO:teuthology.task.internal:Transferring archived files... 2024-03-18T00:16:21.458 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463/remote/smithi012 2024-03-18T00:16:21.458 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T00:16:21.581 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463/remote/smithi079 2024-03-18T00:16:21.581 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T00:16:21.620 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463/remote/smithi191 2024-03-18T00:16:21.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T00:16:21.660 INFO:teuthology.task.internal:Removing archive directory... 2024-03-18T00:16:21.660 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T00:16:21.668 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T00:16:21.670 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T00:16:21.721 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-18T00:16:21.734 INFO:teuthology.task.internal:Not uploading archives. 2024-03-18T00:16:21.734 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-18T00:16:21.804 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-18T00:16:21.805 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T00:16:21.811 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T00:16:21.813 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T00:16:21.825 INFO:teuthology.orchestra.run.smithi079.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 00:16 /home/ubuntu/cephtest 2024-03-18T00:16:21.827 INFO:teuthology.orchestra.run.smithi191.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 00:16 /home/ubuntu/cephtest 2024-03-18T00:16:21.862 INFO:teuthology.orchestra.run.smithi012.stdout: 393943 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 00:16 /home/ubuntu/cephtest 2024-03-18T00:16:21.865 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-18T00:16:21.876 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-18T00:16:21.899 INFO:teuthology.nuke:Checking targets against current locks 2024-03-18T00:16:21.926 DEBUG:teuthology.nuke:shortname: smithi012 2024-03-18T00:16:21.926 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T00:16:21.950 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-17 23:20:11.734961', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGa961Dzqp+tZg2LmYxpfomuSbZshpA1HtJSYIX3CuvgME0iQN+ao9/HIY45qeXnip8dzYNNh0gXIiveDWyCb7Q='} 2024-03-18T00:16:21.954 DEBUG:teuthology.nuke:shortname: smithi079 2024-03-18T00:16:21.954 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T00:16:21.984 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-17 23:20:11.731987', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF13kXw2LFa5gBTd0GORJflZaTdCJwAX+2CxqrDbryHFFpnX5TRr98ypsxSC4IQ6EKun6Sou+S9WFod5Q33Okd4='} 2024-03-18T00:16:21.985 DEBUG:teuthology.nuke:shortname: smithi191 2024-03-18T00:16:21.985 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T00:16:22.013 INFO:teuthology.orchestra.console.smithi012:Power off 2024-03-18T00:16:22.013 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T00:16:22.025 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-17 23:20:11.733405', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKly/HsdlgRfddttfzA8lT1U9Iem8vnkeqm4JoxnTqWbKLFAN+rgu0xjXOGOauZyeTP5ihoV4/mKET/uSXzx7rc='} 2024-03-18T00:16:22.036 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off 2024-03-18T00:16:22.036 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T00:16:22.053 INFO:teuthology.orchestra.console.smithi079:Power off 2024-03-18T00:16:22.054 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T00:16:22.071 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-03-18T00:16:22.076 DEBUG:teuthology.orchestra.console.smithi079:power off output: Chassis Power Control: Down/Off 2024-03-18T00:16:22.076 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T00:16:22.092 INFO:teuthology.orchestra.console.smithi191:Power off 2024-03-18T00:16:22.093 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T00:16:22.100 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-03-18T00:16:22.114 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off 2024-03-18T00:16:22.114 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T00:16:22.140 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-18T00:16:26.072 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T00:16:26.102 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T00:16:26.141 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T00:16:26.184 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-03-18T00:16:26.212 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-03-18T00:16:26.255 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-18T00:16:30.186 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T00:16:30.213 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T00:16:30.255 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T00:16:30.296 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-03-18T00:16:30.324 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is off 2024-03-18T00:16:30.366 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-18T00:16:30.426 INFO:teuthology.orchestra.console.smithi079:Power off completed 2024-03-18T00:16:34.298 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T00:16:34.368 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T00:16:34.410 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off 2024-03-18T00:16:34.479 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off 2024-03-18T00:16:34.510 INFO:teuthology.orchestra.console.smithi012:Power off completed 2024-03-18T00:16:34.580 INFO:teuthology.orchestra.console.smithi191:Power off completed 2024-03-18T00:16:34.686 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 3845b7ac715a3ff198582638d199a90e30e254da description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} duration: 2829.196415424347 failure_reason: '"2024-03-17T23:48:03.070702+0000 mon.a (mon.0) 210 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] is down (out of quorum)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-18T00:16:34.687 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T00:16:34.763 INFO:teuthology.run:FAIL